Nov 03 21:14:02.133152 ip-10-0-128-43 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 21:14:02.566539 ip-10-0-128-43 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 21:14:02.566539 ip-10-0-128-43 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 21:14:02.566539 ip-10-0-128-43 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 21:14:02.566539 ip-10-0-128-43 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 21:14:02.571031 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.570947    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:14:02.571031 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571030    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:14:02.571031 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571033    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:14:02.571031 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571037    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:14:02.571153 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571040    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:14:02.571153 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571042    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:14:02.571153 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571045    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:14:02.571153 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571048    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:14:02.571153 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571050    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:14:02.571153 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571054    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:14:02.571153 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571057    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:14:02.571153 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571059    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:14:02.571153 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571062    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:14:02.571153 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571064    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:14:02.571153 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571067    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:14:02.571153 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571069    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:14:02.571153 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571072    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:14:02.571153 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571074    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:14:02.571153 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571077    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:14:02.571153 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571081    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:14:02.571153 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571084    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:14:02.571153 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571087    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:14:02.571153 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571090    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:14:02.571584 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571092    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:14:02.571584 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571095    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:14:02.571584 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571105    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:14:02.571584 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571108    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:14:02.571584 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571111    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:14:02.571584 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571113    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:14:02.571584 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571116    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:14:02.571584 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571118    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:14:02.571584 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571121    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:14:02.571584 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571123    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:14:02.571584 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571126    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:14:02.571584 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571128    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:14:02.571584 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571131    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:14:02.571584 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571133    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:14:02.571584 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571137    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:14:02.571584 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571140    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:14:02.571584 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571142    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:14:02.571584 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571145    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:14:02.571584 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571148    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:14:02.571584 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571151    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:14:02.572070 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571154    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:14:02.572070 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571156    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:14:02.572070 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571159    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:14:02.572070 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571162    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:14:02.572070 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571165    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:14:02.572070 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571167    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:14:02.572070 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571170    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:14:02.572070 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571172    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:14:02.572070 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571175    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:14:02.572070 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571177    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:14:02.572070 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571179    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:14:02.572070 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571182    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:14:02.572070 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571185    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:14:02.572070 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571187    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:14:02.572070 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571190    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:14:02.572070 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571193    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:14:02.572070 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571195    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:14:02.572070 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571198    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:14:02.572070 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571200    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:14:02.572070 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571203    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:14:02.572623 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571205    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:14:02.572623 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571207    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:14:02.572623 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571210    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:14:02.572623 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571212    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:14:02.572623 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571215    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:14:02.572623 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571217    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:14:02.572623 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571221    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:14:02.572623 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571223    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:14:02.572623 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571226    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:14:02.572623 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571229    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:14:02.572623 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571231    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:14:02.572623 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571234    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:14:02.572623 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571236    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:14:02.572623 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571239    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:14:02.572623 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571242    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:14:02.572623 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571244    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:14:02.572623 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571247    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:14:02.572623 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571249    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:14:02.572623 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571252    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:14:02.573096 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571255    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:14:02.573096 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571258    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:14:02.573096 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571260    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:14:02.573096 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571262    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:14:02.573096 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571265    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:14:02.573096 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571267    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:14:02.573096 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571271    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:14:02.573096 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571274    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:14:02.573096 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571276    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:14:02.573096 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571279    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:14:02.573096 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571281    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:14:02.573096 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571284    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:14:02.573096 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571286    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:14:02.573096 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571289    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:14:02.573096 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571291    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:14:02.573096 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571295    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:14:02.573096 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571299    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:14:02.573096 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571301    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:14:02.573096 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571304    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:14:02.573534 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571308    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:14:02.573534 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571311    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:14:02.573534 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571314    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:14:02.573534 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571317    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:14:02.573534 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.571321    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:14:02.573534 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572158    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:14:02.573534 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572165    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:14:02.573534 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572169    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:14:02.573534 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572172    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:14:02.573534 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572175    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:14:02.573534 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572177    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:14:02.573534 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572180    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:14:02.573534 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572183    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:14:02.573534 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572185    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:14:02.573534 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572188    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:14:02.573534 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572190    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:14:02.573534 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572193    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:14:02.573534 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572195    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:14:02.573534 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572198    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:14:02.573534 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572200    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:14:02.574039 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572203    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:14:02.574039 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572205    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:14:02.574039 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572208    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:14:02.574039 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572212    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:14:02.574039 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572215    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:14:02.574039 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572218    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:14:02.574039 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572221    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:14:02.574039 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572223    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:14:02.574039 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572226    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:14:02.574039 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572229    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:14:02.574039 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572231    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:14:02.574039 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572234    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:14:02.574039 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572238    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:14:02.574039 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572240    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:14:02.574039 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572243    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:14:02.574039 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572245    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:14:02.574039 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572248    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:14:02.574039 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572250    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:14:02.574039 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572253    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:14:02.574486 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572255    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:14:02.574486 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572257    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:14:02.574486 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572261    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:14:02.574486 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572264    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:14:02.574486 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572266    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:14:02.574486 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572269    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:14:02.574486 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572271    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:14:02.574486 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572273    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:14:02.574486 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572276    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:14:02.574486 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572278    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:14:02.574486 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572281    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:14:02.574486 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572283    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:14:02.574486 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572286    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:14:02.574486 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572288    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:14:02.574486 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572291    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:14:02.574486 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572293    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:14:02.574486 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572296    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:14:02.574486 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572299    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:14:02.574486 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572301    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:14:02.574951 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572304    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:14:02.574951 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572306    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:14:02.574951 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572310    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:14:02.574951 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572314    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:14:02.574951 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572316    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:14:02.574951 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572319    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:14:02.574951 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572322    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:14:02.574951 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572325    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:14:02.574951 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572328    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:14:02.574951 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572330    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:14:02.574951 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572332    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:14:02.574951 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572335    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:14:02.574951 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572343    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:14:02.574951 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572346    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:14:02.574951 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572348    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:14:02.574951 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572351    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:14:02.574951 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572354    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:14:02.574951 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572356    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:14:02.574951 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572359    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:14:02.575404 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572361    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:14:02.575404 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572363    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:14:02.575404 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572366    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:14:02.575404 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572368    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:14:02.575404 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572371    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:14:02.575404 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572373    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:14:02.575404 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572376    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:14:02.575404 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572378    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:14:02.575404 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572381    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:14:02.575404 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572383    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:14:02.575404 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572386    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:14:02.575404 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572388    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:14:02.575404 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572391    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:14:02.575404 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572393    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:14:02.575404 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572396    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:14:02.575404 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572399    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:14:02.575404 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572401    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:14:02.575404 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572403    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:14:02.575404 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572406    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:14:02.575404 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572409    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:14:02.575921 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572411    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:14:02.575921 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572414    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:14:02.575921 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572417    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:14:02.575921 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572419    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:14:02.575921 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572422    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:14:02.575921 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572425    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:14:02.575921 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572427    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:14:02.575921 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572429    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:14:02.575921 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572432    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:14:02.575921 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572434    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:14:02.575921 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572437    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:14:02.575921 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572440    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:14:02.575921 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572442    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:14:02.575921 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572446    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:14:02.575921 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572956    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:14:02.575921 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572962    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:14:02.575921 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572964    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:14:02.575921 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572967    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:14:02.575921 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572970    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:14:02.575921 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572972    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:14:02.576385 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572975    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:14:02.576385 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572978    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:14:02.576385 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572980    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:14:02.576385 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572983    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:14:02.576385 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572985    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:14:02.576385 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572988    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:14:02.576385 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572990    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:14:02.576385 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572995    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:14:02.576385 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.572998    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:14:02.576385 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573001    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:14:02.576385 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573004    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:14:02.576385 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573007    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:14:02.576385 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573010    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:14:02.576385 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573012    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:14:02.576385 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573014    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:14:02.576385 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573017    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:14:02.576385 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573019    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:14:02.576385 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573022    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:14:02.576385 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573024    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:14:02.576831 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573028    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:14:02.576831 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573031    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:14:02.576831 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573035    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:14:02.576831 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573039    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:14:02.576831 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573042    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:14:02.576831 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573044    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:14:02.576831 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573047    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:14:02.576831 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573049    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:14:02.576831 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573052    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:14:02.576831 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573054    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:14:02.576831 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573056    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:14:02.576831 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573058    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:14:02.576831 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573061    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:14:02.576831 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573063    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:14:02.576831 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573066    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:14:02.576831 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573068    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:14:02.576831 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573070    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:14:02.576831 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573073    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:14:02.576831 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573075    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:14:02.577320 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573077    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:14:02.577320 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573080    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:14:02.577320 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573082    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:14:02.577320 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573085    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:14:02.577320 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573088    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:14:02.577320 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573090    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:14:02.577320 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573093    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:14:02.577320 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573095    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:14:02.577320 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573098    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:14:02.577320 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573100    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:14:02.577320 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573103    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:14:02.577320 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573106    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:14:02.577320 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573108    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:14:02.577320 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573111    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:14:02.577320 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573113    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:14:02.577320 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573115    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:14:02.577320 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573118    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:14:02.577320 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573121    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:14:02.577320 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573123    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:14:02.577320 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573126    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:14:02.577782 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573128    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:14:02.577782 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573131    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:14:02.577782 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573134    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:14:02.577782 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573137    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:14:02.577782 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573139    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:14:02.577782 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573142    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:14:02.577782 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573144    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:14:02.577782 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573147    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:14:02.577782 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573149    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:14:02.577782 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573152    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:14:02.577782 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573154    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:14:02.577782 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573157    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:14:02.577782 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573159    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:14:02.577782 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573162    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:14:02.577782 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573164    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:14:02.577782 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573166    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:14:02.577782 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573169    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:14:02.577782 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573172    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:14:02.577782 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573175    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:14:02.578246 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573178    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:14:02.578246 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573181    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:14:02.578246 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573183    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:14:02.578246 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573186    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:14:02.578246 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573188    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:14:02.578246 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573191    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:14:02.578246 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573193    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:14:02.578246 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573196    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:14:02.578246 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573198    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:14:02.578246 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573200    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:14:02.578246 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573203    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:14:02.578246 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573206    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:14:02.578246 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573208    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:14:02.578246 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573211    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:14:02.578246 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573213    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:14:02.578246 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573215    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:14:02.578246 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573218    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:14:02.578246 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573220    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:14:02.578246 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573223    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:14:02.578246 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573226    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:14:02.578739 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573228    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:14:02.578739 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573231    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:14:02.578739 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.573234    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:14:02.578739 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574182    2574 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 21:14:02.578739 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574192    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 21:14:02.578739 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574198    2574 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 21:14:02.578739 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574203    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 21:14:02.578739 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574208    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 21:14:02.578739 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574211    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 21:14:02.578739 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574216    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 21:14:02.578739 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574220    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 21:14:02.578739 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574225    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 21:14:02.578739 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574228    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 21:14:02.578739 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574231    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 21:14:02.578739 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574234    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 21:14:02.578739 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574237    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 21:14:02.578739 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574240    2574 flags.go:64] FLAG: --cgroup-root=""
Nov 03 21:14:02.578739 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574243    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 21:14:02.578739 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574246    2574 flags.go:64] FLAG: --client-ca-file=""
Nov 03 21:14:02.578739 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574249    2574 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 21:14:02.578739 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574252    2574 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 21:14:02.578739 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574256    2574 flags.go:64] FLAG: --cluster-domain=""
Nov 03 21:14:02.578739 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574259    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 21:14:02.579280 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574263    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 21:14:02.579280 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574267    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 21:14:02.579280 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574270    2574 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 21:14:02.579280 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574274    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 21:14:02.579280 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574277    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 21:14:02.579280 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574280    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 21:14:02.579280 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574283    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 21:14:02.579280 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574286    2574 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 21:14:02.579280 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574289    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 21:14:02.579280 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574292    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 21:14:02.579280 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574295    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 21:14:02.579280 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574300    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 21:14:02.579280 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574304    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 21:14:02.579280 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574307    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 21:14:02.579280 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574310    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 21:14:02.579280 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574313    2574 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 21:14:02.579280 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574316    2574 flags.go:64] FLAG: --enable-server="true"
Nov 03 21:14:02.579280 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574319    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 21:14:02.579280 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574323    2574 flags.go:64] FLAG: --event-burst="100"
Nov 03 21:14:02.579280 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574329    2574 flags.go:64] FLAG: --event-qps="50"
Nov 03 21:14:02.579280 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574332    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 21:14:02.579280 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574335    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 21:14:02.579280 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574338    2574 flags.go:64] FLAG: --eviction-hard=""
Nov 03 21:14:02.579280 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574342    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 21:14:02.579854 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574345    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 21:14:02.579854 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574348    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 21:14:02.579854 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574351    2574 flags.go:64] FLAG: --eviction-soft=""
Nov 03 21:14:02.579854 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574354    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 21:14:02.579854 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574357    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 21:14:02.579854 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574360    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 21:14:02.579854 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574363    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 21:14:02.579854 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574366    2574 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 21:14:02.579854 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574369    2574 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 21:14:02.579854 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574371    2574 flags.go:64] FLAG: --feature-gates=""
Nov 03 21:14:02.579854 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574376    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 21:14:02.579854 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574379    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 21:14:02.579854 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574382    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 21:14:02.579854 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574385    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 21:14:02.579854 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574389    2574 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 21:14:02.579854 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574392    2574 flags.go:64] FLAG: --help="false"
Nov 03 21:14:02.579854 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574394    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:02.579854 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574398    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 21:14:02.579854 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574401    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 21:14:02.579854 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574405    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 21:14:02.579854 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574408    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 21:14:02.579854 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574411    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 21:14:02.579854 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574414    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574417    2574 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574420    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574423    2574 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574425    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574430    2574 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574433    2574 flags.go:64] FLAG: --kube-reserved=""
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574435    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574438    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574441    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574444    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574447    2574 flags.go:64] FLAG: --lock-file=""
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574450    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574453    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574455    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574461    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574464    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574467    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574470    2574 flags.go:64] FLAG: --logging-format="text"
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574472    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574476    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574479    2574 flags.go:64] FLAG: --manifest-url=""
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574482    2574 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574486    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574489    2574 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 21:14:02.580417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574493    2574 flags.go:64] FLAG: --max-pods="110"
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574497    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574500    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574503    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574507    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574510    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574513    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574516    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574523    2574 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574527    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574529    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574532    2574 flags.go:64] FLAG: --pod-cidr=""
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574536    2574 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574539    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574542    2574 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574545    2574 flags.go:64] FLAG: --port="10250"
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574548    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574551    2574 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-044a84e5a53399cbf"
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574555    2574 flags.go:64] FLAG: --qos-reserved=""
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574557    2574 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574560    2574 flags.go:64] FLAG: --register-node="true"
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574563    2574 flags.go:64] FLAG: --register-with-taints=""
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574567    2574 flags.go:64] FLAG: --registry-burst="10"
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574570    2574 flags.go:64] FLAG: --registry-qps="5"
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574572    2574 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574575    2574 flags.go:64] FLAG: --reserved-memory=""
Nov 03 21:14:02.581009 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574579    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574582    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574586    2574 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574589    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574592    2574 flags.go:64] FLAG: --runonce="false"
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574595    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574598    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574600    2574 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574603    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574606    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574610    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574614    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574617    2574 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574620    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574623    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574626    2574 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574628    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574631    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574635    2574 flags.go:64] FLAG: --system-cgroups=""
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574638    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574643    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574646    2574 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574649    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574653    2574 flags.go:64] FLAG: --tls-min-version=""
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574656    2574 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 21:14:02.581611 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574659    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 21:14:02.582248 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574662    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 21:14:02.582248 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574665    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 21:14:02.582248 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574668    2574 flags.go:64] FLAG: --v="2"
Nov 03 21:14:02.582248 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574672    2574 flags.go:64] FLAG: --version="false"
Nov 03 21:14:02.582248 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574676    2574 flags.go:64] FLAG: --vmodule=""
Nov 03 21:14:02.582248 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574680    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 21:14:02.582248 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.574683    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 21:14:02.582248 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574822    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:14:02.582248 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574826    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:14:02.582248 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574829    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:14:02.582248 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574833    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:14:02.582248 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574836    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:14:02.582248 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574838    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:14:02.582248 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574841    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:14:02.582248 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574843    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:14:02.582248 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574845    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:14:02.582248 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574849    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:14:02.582248 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574852    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:14:02.582248 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574854    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:14:02.582248 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574869    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:14:02.582248 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574871    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:14:02.582734 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574874    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:14:02.582734 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574876    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:14:02.582734 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574879    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:14:02.582734 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574883    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:14:02.582734 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574886    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:14:02.582734 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574888    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:14:02.582734 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574891    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:14:02.582734 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574893    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:14:02.582734 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574896    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:14:02.582734 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574899    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:14:02.582734 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574901    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:14:02.582734 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574903    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:14:02.582734 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574908    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:14:02.582734 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574912    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:14:02.582734 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574916    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:14:02.582734 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574918    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:14:02.582734 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574921    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:14:02.582734 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574924    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:14:02.582734 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574926    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:14:02.583197 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574930    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:14:02.583197 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574934    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:14:02.583197 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574937    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:14:02.583197 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574940    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:14:02.583197 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574943    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:14:02.583197 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574946    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:14:02.583197 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574948    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:14:02.583197 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574951    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:14:02.583197 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574955    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:14:02.583197 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574957    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:14:02.583197 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574960    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:14:02.583197 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574962    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:14:02.583197 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574965    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:14:02.583197 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574967    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:14:02.583197 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574970    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:14:02.583197 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574972    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:14:02.583197 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574976    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:14:02.583197 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574979    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:14:02.583197 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574982    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:14:02.583197 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574984    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:14:02.583689 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574987    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:14:02.583689 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574989    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:14:02.583689 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574992    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:14:02.583689 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574994    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:14:02.583689 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574997    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:14:02.583689 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.574999    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:14:02.583689 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575001    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:14:02.583689 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575004    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:14:02.583689 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575006    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:14:02.583689 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575009    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:14:02.583689 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575011    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:14:02.583689 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575013    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:14:02.583689 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575016    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:14:02.583689 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575018    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:14:02.583689 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575021    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:14:02.583689 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575023    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:14:02.583689 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575026    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:14:02.583689 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575029    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:14:02.583689 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575032    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:14:02.583689 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575034    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:14:02.584164 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575037    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:14:02.584164 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575040    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:14:02.584164 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575042    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:14:02.584164 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575044    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:14:02.584164 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575047    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:14:02.584164 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575049    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:14:02.584164 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575052    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:14:02.584164 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575054    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:14:02.584164 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575058    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:14:02.584164 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575060    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:14:02.584164 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575063    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:14:02.584164 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575065    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:14:02.584164 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575068    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:14:02.584164 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575070    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:14:02.584164 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575073    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:14:02.584164 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575075    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:14:02.584164 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575078    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:14:02.584164 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575080    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:14:02.584164 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575082    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:14:02.584164 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575085    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:14:02.584630 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575087    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:14:02.584630 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575090    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:14:02.584630 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575092    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:14:02.584630 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575095    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:14:02.584630 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575097    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:14:02.584630 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575099    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:14:02.584630 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575102    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:14:02.584630 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575104    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:14:02.584630 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575107    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:14:02.584630 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575110    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:14:02.584630 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575112    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:14:02.584630 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575115    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:14:02.584630 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575117    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:14:02.584630 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.575144    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 21:14:02.584630 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575208    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:14:02.584630 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575213    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:14:02.585017 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575217    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:14:02.585017 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575219    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:14:02.585017 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575222    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:14:02.585017 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575225    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:14:02.585017 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575227    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:14:02.585017 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575231    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:14:02.585017 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575234    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:14:02.585017 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575237    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:14:02.585017 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575241    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:14:02.585017 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575244    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:14:02.585017 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575247    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:14:02.585017 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575249    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:14:02.585017 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575252    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:14:02.585017 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575254    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:14:02.585017 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575258    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:14:02.585017 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575261    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:14:02.585017 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575264    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:14:02.585017 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575267    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:14:02.585017 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575270    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:14:02.585468 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575273    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:14:02.585468 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575275    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:14:02.585468 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575278    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:14:02.585468 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575280    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:14:02.585468 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575283    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:14:02.585468 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575285    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:14:02.585468 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575287    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:14:02.585468 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575293    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:14:02.585468 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575296    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:14:02.585468 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575299    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:14:02.585468 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575301    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:14:02.585468 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575304    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:14:02.585468 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575307    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:14:02.585468 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575309    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:14:02.585468 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575312    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:14:02.585468 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575314    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:14:02.585468 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575316    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:14:02.585468 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575318    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:14:02.585468 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575322    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:14:02.585468 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575326    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:14:02.585952 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575328    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:14:02.585952 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575331    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:14:02.585952 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575333    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:14:02.585952 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575336    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:14:02.585952 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575338    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:14:02.585952 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575340    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:14:02.585952 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575343    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:14:02.585952 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575345    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:14:02.585952 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575348    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:14:02.585952 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575350    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:14:02.585952 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575353    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:14:02.585952 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575355    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:14:02.585952 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575357    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:14:02.585952 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575360    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:14:02.585952 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575362    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:14:02.585952 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575365    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:14:02.585952 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575367    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:14:02.585952 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575370    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:14:02.585952 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575372    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:14:02.585952 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575375    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:14:02.586406 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575378    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:14:02.586406 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575380    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:14:02.586406 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575383    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:14:02.586406 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575385    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:14:02.586406 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575388    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:14:02.586406 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575390    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:14:02.586406 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575393    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:14:02.586406 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575395    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:14:02.586406 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575397    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:14:02.586406 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575399    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:14:02.586406 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575403    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:14:02.586406 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575406    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:14:02.586406 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575409    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:14:02.586406 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575411    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:14:02.586406 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575414    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:14:02.586406 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575417    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:14:02.586406 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575419    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:14:02.586406 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575421    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:14:02.586406 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575424    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:14:02.586837 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575426    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:14:02.586837 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575429    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:14:02.586837 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575432    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:14:02.586837 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575434    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:14:02.586837 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575436    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:14:02.586837 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575439    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:14:02.586837 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575441    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:14:02.586837 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575444    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:14:02.586837 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575446    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:14:02.586837 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575449    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:14:02.586837 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575451    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:14:02.586837 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575454    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:14:02.586837 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575457    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:14:02.586837 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575459    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:14:02.586837 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575464    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:14:02.586837 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575466    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:14:02.586837 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575469    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:14:02.586837 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575471    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:14:02.586837 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575474    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:14:02.587335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575476    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:14:02.587335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575478    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:14:02.587335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575481    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:14:02.587335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575483    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:14:02.587335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575486    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:14:02.587335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575489    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:14:02.587335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.575491    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:14:02.587335 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.583087    2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 21:14:02.587335 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.583103    2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 21:14:02.587335 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.583786    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 21:14:02.587335 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.583791    2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 21:14:02.587335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583873    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:14:02.587335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583879    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:14:02.587335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583882    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:14:02.587335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583886    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:14:02.587335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583888    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:14:02.587335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583891    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:14:02.587335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583894    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:14:02.587335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583896    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:14:02.587785 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583899    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:14:02.587785 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583902    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:14:02.587785 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583904    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:14:02.587785 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583907    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:14:02.587785 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583909    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:14:02.587785 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583912    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:14:02.587785 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583914    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:14:02.587785 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583917    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:14:02.587785 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583919    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:14:02.587785 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583922    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:14:02.587785 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583925    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:14:02.587785 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583927    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:14:02.587785 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583930    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:14:02.587785 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583933    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:14:02.587785 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583935    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:14:02.587785 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583938    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:14:02.587785 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583940    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:14:02.587785 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583943    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:14:02.587785 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583945    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:14:02.588335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583950    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:14:02.588335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583954    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:14:02.588335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583963    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:14:02.588335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583967    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:14:02.588335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583969    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:14:02.588335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583972    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:14:02.588335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583975    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:14:02.588335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583977    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:14:02.588335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583980    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:14:02.588335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583982    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:14:02.588335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583985    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:14:02.588335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583987    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:14:02.588335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583990    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:14:02.588335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583992    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:14:02.588335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583995    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:14:02.588335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.583997    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:14:02.588335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584000    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:14:02.588335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584012    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:14:02.588335 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584015    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:14:02.588775 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584017    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:14:02.588775 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584020    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:14:02.588775 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584022    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:14:02.588775 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584025    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:14:02.588775 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584027    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:14:02.588775 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584030    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:14:02.588775 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584032    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:14:02.588775 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584035    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:14:02.588775 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584037    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:14:02.588775 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584040    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:14:02.588775 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584042    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:14:02.588775 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584045    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:14:02.588775 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584047    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:14:02.588775 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584050    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:14:02.588775 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584053    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:14:02.588775 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584056    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:14:02.588775 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584064    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:14:02.588775 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584067    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:14:02.588775 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584070    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:14:02.588775 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584072    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:14:02.589243 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584074    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:14:02.589243 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584077    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:14:02.589243 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584079    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:14:02.589243 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584082    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:14:02.589243 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584084    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:14:02.589243 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584087    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:14:02.589243 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584090    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:14:02.589243 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584094    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:14:02.589243 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584098    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:14:02.589243 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584101    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:14:02.589243 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584104    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:14:02.589243 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584106    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:14:02.589243 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584109    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:14:02.589243 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584111    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:14:02.589243 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584114    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:14:02.589243 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584116    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:14:02.589243 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584119    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:14:02.589243 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584121    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:14:02.589243 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584124    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:14:02.589708 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584126    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:14:02.589708 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584128    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:14:02.589708 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584131    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:14:02.589708 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584133    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:14:02.589708 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584136    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:14:02.589708 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584138    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:14:02.589708 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584141    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:14:02.589708 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584144    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:14:02.589708 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584146    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:14:02.589708 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584149    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:14:02.589708 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584157    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:14:02.589708 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584159    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:14:02.589708 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584162    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:14:02.589708 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584164    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:14:02.589708 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584167    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:14:02.589708 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584169    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:14:02.589708 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584172    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:14:02.589708 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584174    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:14:02.589708 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584177    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:14:02.589708 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584179    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:14:02.590187 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584182    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:14:02.590187 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.584209    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 21:14:02.590187 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584403    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:14:02.590187 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584407    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:14:02.590187 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584410    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:14:02.590187 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584413    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:14:02.590187 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584416    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:14:02.590187 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584418    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:14:02.590187 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584421    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:14:02.590187 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584423    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:14:02.590187 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584426    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:14:02.590187 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584428    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:14:02.590187 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584431    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:14:02.590187 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584433    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:14:02.590187 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584436    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:14:02.590187 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584438    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:14:02.590561 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584441    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:14:02.590561 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584443    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:14:02.590561 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584446    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:14:02.590561 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584448    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:14:02.590561 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584451    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:14:02.590561 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584453    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:14:02.590561 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584455    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:14:02.590561 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584463    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:14:02.590561 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584466    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:14:02.590561 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584468    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:14:02.590561 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584471    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:14:02.590561 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584473    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:14:02.590561 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584476    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:14:02.590561 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584478    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:14:02.590561 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584480    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:14:02.590561 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584483    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:14:02.590561 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584485    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:14:02.590561 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584487    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:14:02.590561 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584490    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:14:02.591016 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584492    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:14:02.591016 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584495    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:14:02.591016 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584497    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:14:02.591016 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584500    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:14:02.591016 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584502    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:14:02.591016 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584504    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:14:02.591016 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584507    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:14:02.591016 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584509    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:14:02.591016 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584512    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:14:02.591016 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584514    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:14:02.591016 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584517    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:14:02.591016 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584519    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:14:02.591016 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584522    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:14:02.591016 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584524    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:14:02.591016 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584528    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:14:02.591016 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584531    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:14:02.591016 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584534    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:14:02.591016 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584537    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:14:02.591016 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584540    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:14:02.591451 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584544    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:14:02.591451 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584547    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:14:02.591451 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584555    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:14:02.591451 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584558    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:14:02.591451 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584560    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:14:02.591451 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584563    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:14:02.591451 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584565    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:14:02.591451 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584568    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:14:02.591451 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584570    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:14:02.591451 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584572    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:14:02.591451 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584575    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:14:02.591451 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584577    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:14:02.591451 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584579    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:14:02.591451 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584582    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:14:02.591451 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584584    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:14:02.591451 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584587    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:14:02.591451 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584589    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:14:02.591451 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584591    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:14:02.591451 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584594    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:14:02.591451 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584596    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:14:02.592625 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584598    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:14:02.592625 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584601    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:14:02.592625 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584603    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:14:02.592625 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584606    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:14:02.592625 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584608    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:14:02.592625 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584610    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:14:02.592625 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584613    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:14:02.592625 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584615    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:14:02.592625 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584618    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:14:02.592625 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584620    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:14:02.592625 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584622    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:14:02.592625 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584624    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:14:02.592625 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584627    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:14:02.592625 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584629    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:14:02.592625 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584631    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:14:02.592625 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584639    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:14:02.592625 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584642    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:14:02.592625 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584645    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:14:02.592625 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584647    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:14:02.592625 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584649    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:14:02.593347 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584652    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:14:02.593347 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584654    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:14:02.593347 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584656    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:14:02.593347 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584658    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:14:02.593347 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584661    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:14:02.593347 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584663    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:14:02.593347 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584665    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:14:02.593347 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584668    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:14:02.593347 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584670    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:14:02.593347 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584672    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:14:02.593347 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584675    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:14:02.593347 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584677    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:14:02.593347 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584680    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:14:02.593347 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584682    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:14:02.593347 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.584705    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 21:14:02.593347 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584747    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:14:02.593826 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584752    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:14:02.593826 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584755    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:14:02.593826 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584757    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:14:02.593826 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584760    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:14:02.593826 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584763    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:14:02.593826 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584765    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:14:02.593826 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584768    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:14:02.593826 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584770    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:14:02.593826 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584773    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:14:02.593826 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584775    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:14:02.593826 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584777    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:14:02.593826 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584780    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:14:02.593826 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584788    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:14:02.593826 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584790    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:14:02.593826 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584793    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:14:02.593826 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584795    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:14:02.593826 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584797    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:14:02.593826 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584800    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:14:02.593826 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584802    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:14:02.593826 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584805    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:14:02.594428 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584807    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:14:02.594428 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584809    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:14:02.594428 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584812    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:14:02.594428 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584814    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:14:02.594428 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584816    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:14:02.594428 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584819    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:14:02.594428 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584821    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:14:02.594428 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584823    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:14:02.594428 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584826    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:14:02.594428 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584828    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:14:02.594428 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584831    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:14:02.594428 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584833    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:14:02.594428 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584836    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:14:02.594428 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584838    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:14:02.594428 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584841    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:14:02.594428 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584844    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:14:02.594428 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584849    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:14:02.594428 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584852    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:14:02.594428 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584855    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:14:02.594428 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584870    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:14:02.595058 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584874    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:14:02.595058 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584878    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:14:02.595058 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584882    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:14:02.595058 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584886    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:14:02.595058 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584889    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:14:02.595058 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584899    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:14:02.595058 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584902    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:14:02.595058 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584905    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:14:02.595058 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584907    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:14:02.595058 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584910    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:14:02.595058 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584912    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:14:02.595058 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584915    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:14:02.595058 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584917    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:14:02.595058 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584925    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:14:02.595058 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584928    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:14:02.595058 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584930    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:14:02.595058 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584933    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:14:02.595058 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584935    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:14:02.595058 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584938    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:14:02.595492 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584940    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:14:02.595492 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584943    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:14:02.595492 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584947    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:14:02.595492 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584949    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:14:02.595492 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584953    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:14:02.595492 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584956    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:14:02.595492 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584958    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:14:02.595492 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584961    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:14:02.595492 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584963    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:14:02.595492 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584966    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:14:02.595492 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584968    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:14:02.595492 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584971    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:14:02.595492 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584973    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:14:02.595492 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584976    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:14:02.595492 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584978    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:14:02.595492 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584981    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:14:02.595492 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584984    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:14:02.595492 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584986    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:14:02.595492 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.584990    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:14:02.595982 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585000    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:14:02.595982 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585003    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:14:02.595982 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585006    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:14:02.595982 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585010    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:14:02.595982 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585014    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:14:02.595982 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585017    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:14:02.595982 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585020    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:14:02.595982 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585023    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:14:02.595982 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585025    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:14:02.595982 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585028    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:14:02.595982 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585030    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:14:02.595982 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585033    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:14:02.595982 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585035    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:14:02.595982 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585038    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:14:02.595982 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585041    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:14:02.595982 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585045    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:14:02.595982 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585048    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:14:02.595982 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585051    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:14:02.595982 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585053    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:14:02.596645 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585056    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:14:02.596645 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585058    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:14:02.596645 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585061    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:14:02.596645 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585063    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:14:02.596645 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585065    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:14:02.596645 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585067    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:14:02.596645 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585070    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:14:02.596645 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:02.585072    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:14:02.596645 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.585136    2574 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 21:14:02.596645 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.587523    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 21:14:02.596645 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.588494    2574 server.go:1014] "Starting client certificate rotation"
Nov 03 21:14:02.596645 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.588602    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 21:14:02.596645 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.588634    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 21:14:02.596645 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.590778    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 21:14:02.596645 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.594514    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 21:14:02.605085 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.605067    2574 log.go:25] "Validated CRI v1 runtime API"
Nov 03 21:14:02.609030 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.609015    2574 log.go:25] "Validated CRI v1 image API"
Nov 03 21:14:02.609918 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.609884    2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 21:14:02.616090 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.616073    2574 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ba4d75bf-5393-4569-bdef-490af2af4b7e:/dev/nvme0n1p3 c0aae208-0c17-4504-90a8-de75bb41017d:/dev/nvme0n1p4]
Nov 03 21:14:02.616137 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.616091    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 21:14:02.620383 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.620276    2574 manager.go:217] Machine: {Timestamp:2026-11-03 21:14:02.619776813 +0000 UTC m=+0.361778836 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3137125 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bcb1fce5030842ca50888659c98d1 SystemUUID:ec2bcb1f-ce50-3084-2ca5-0888659c98d1 BootID:60e91c6b-2423-4717-bd4a-369f194e7ccb Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0d:50:7a:09:f3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0d:50:7a:09:f3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:17:7f:68:38:23 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 21:14:02.620383 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.620378    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 21:14:02.620501 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.620458    2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 21:14:02.624370 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.624341    2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 21:14:02.624540 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.624372    2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-43.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 21:14:02.624617 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.624544    2574 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 21:14:02.624617 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.624557    2574 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 21:14:02.624617 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.624578    2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 21:14:02.624617 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.624603    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 21:14:02.625793 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.625780    2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 21:14:02.630325 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.630310    2574 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 21:14:02.630477 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.630465    2574 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 21:14:02.631301 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.631290    2574 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 21:14:02.631354 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.631308    2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 21:14:02.631354 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.631321    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 21:14:02.631354 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.631335    2574 kubelet.go:405] "Adding apiserver pod source"
Nov 03 21:14:02.631354 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.631347    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 21:14:02.631750 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.631736    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 21:14:02.634103 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.634085    2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 21:14:02.641129 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.641114    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 21:14:02.642045 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.642033    2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 21:14:02.642089 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.642063    2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 21:14:02.642221 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.642211    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 21:14:02.642253 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.642228    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 21:14:02.642253 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.642234    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 21:14:02.642253 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.642239    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 21:14:02.642253 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.642245    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 21:14:02.642253 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.642251    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 21:14:02.642994 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.642985    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 21:14:02.642994 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.642995    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 21:14:02.643047 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.643002    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 21:14:02.643047 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.643007    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 21:14:02.643047 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.643015    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 21:14:02.643047 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.643024    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 21:14:02.643945 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.643935    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 21:14:02.643945 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.643945    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 21:14:02.647918 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.647905    2574 server.go:1263] "Started kubelet"
Nov 03 21:14:02.648726 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.648689    2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 21:14:02.649161 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.648674    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 21:14:02.649245 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.649190    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 21:14:02.649427 ip-10-0-128-43 systemd[1]: Started Kubernetes Kubelet.
Nov 03 21:14:02.650710 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.650695    2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 21:14:02.653645 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.653625    2574 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 21:14:02.657336 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:02.657317    2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 21:14:02.659476 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.659457    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 21:14:02.660163 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.660047    2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 21:14:02.660217 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:02.660160    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-43.us-west-1.compute.internal\" not found"
Nov 03 21:14:02.660923 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.660902    2574 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 21:14:02.661011 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.660925    2574 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 21:14:02.661063 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.661047    2574 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 21:14:02.661063 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.661056    2574 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 21:14:02.661150 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.661086    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 21:14:02.661268 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.661255    2574 factory.go:55] Registering systemd factory
Nov 03 21:14:02.661302 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.661271    2574 factory.go:223] Registration of the systemd container factory successfully
Nov 03 21:14:02.661480 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.661467    2574 factory.go:153] Registering CRI-O factory
Nov 03 21:14:02.661516 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.661485    2574 factory.go:223] Registration of the crio container factory successfully
Nov 03 21:14:02.661543 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.661536    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 21:14:02.661577 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.661556    2574 factory.go:103] Registering Raw factory
Nov 03 21:14:02.661577 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.661573    2574 manager.go:1199] Started watching for new ooms in manager
Nov 03 21:14:02.662630 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.662614    2574 manager.go:319] Starting recovery of all containers
Nov 03 21:14:02.672097 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.672062    2574 manager.go:324] Recovery completed
Nov 03 21:14:02.676094 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.676078    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 21:14:02.677745 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.677717    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 21:14:02.677816 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.677762    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 21:14:02.677816 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.677774    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 21:14:02.678338 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.678327    2574 cpu_manager.go:226] "Starting" policy="none"
Nov 03 21:14:02.678386 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.678337    2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 21:14:02.678386 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.678355    2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 21:14:02.678442 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.678332    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 21:14:02.679070 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.679056    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 21:14:02.679123 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.679087    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 21:14:02.679123 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.679098    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 21:14:02.680564 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.680553    2574 policy_none.go:50] "Start"
Nov 03 21:14:02.680599 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.680569    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 21:14:02.680599 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.680580    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 21:14:02.682615 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.682605    2574 policy_none.go:44] "Start"
Nov 03 21:14:02.746471 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.731498    2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 21:14:02.746471 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.735759    2574 manager.go:341] "Starting Device Plugin manager"
Nov 03 21:14:02.746471 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:02.735789    2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 21:14:02.746471 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.735799    2574 server.go:87] "Starting device plugin registration server"
Nov 03 21:14:02.746471 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.736095    2574 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 21:14:02.746471 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.736109    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 21:14:02.746471 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.736186    2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 21:14:02.746471 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.736258    2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 21:14:02.746471 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.736265    2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 21:14:02.746471 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:02.736817    2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 21:14:02.746471 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:02.736891    2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-43.us-west-1.compute.internal\" not found"
Nov 03 21:14:02.746471 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.741711    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 21:14:02.746471 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.743046    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 21:14:02.746471 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.743066    2574 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 21:14:02.746471 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.743084    2574 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 21:14:02.746471 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:02.743117    2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 21:14:02.790243 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:02.790205    2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-43.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 21:14:02.790243 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.790229    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-43.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 21:14:02.791295 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:02.790222    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-43.us-west-1.compute.internal.18e4a56dcf44369b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-43.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-43.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 21:14:02.647672475 +0000 UTC m=+0.389674497,LastTimestamp:2026-11-03 21:14:02.647672475 +0000 UTC m=+0.389674497,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-43.us-west-1.compute.internal,}"
Nov 03 21:14:02.836662 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.836597    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 21:14:02.838834 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.838810    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 21:14:02.838996 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.838849    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 21:14:02.838996 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.838893    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 21:14:02.838996 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.838925    2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:02.843479 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.843460    2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal"]
Nov 03 21:14:02.843570 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.843523    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 21:14:02.844372 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.844349    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 21:14:02.844445 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.844379    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 21:14:02.844445 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.844390    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 21:14:02.846610 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.846597    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 21:14:02.846765 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.846738    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:02.846810 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.846781    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 21:14:02.847523 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.847505    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 21:14:02.847523 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.847516    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 21:14:02.847622 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.847539    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 21:14:02.847622 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.847554    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 21:14:02.847622 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.847540    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 21:14:02.847622 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.847614    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 21:14:02.849516 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.849499    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:02.849610 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.849523    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 21:14:02.850983 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.850968    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 21:14:02.851068 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.850993    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 21:14:02.851068 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.851004    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 21:14:02.862370 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.862347    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/104317fa8cfa5db2df29de73ade6b849-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal\" (UID: \"104317fa8cfa5db2df29de73ade6b849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:02.862447 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.862371    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/104317fa8cfa5db2df29de73ade6b849-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal\" (UID: \"104317fa8cfa5db2df29de73ade6b849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:02.862447 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.862387    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8253f9611b0fc9c6ff64af8baa405814-config\") pod \"kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal\" (UID: \"8253f9611b0fc9c6ff64af8baa405814\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:02.868502 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.868483    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 21:14:02.868552 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.868539    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5w8dr"
Nov 03 21:14:02.883412 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:02.883395    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-43.us-west-1.compute.internal\" not found" node="ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:02.885369 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:02.885352    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-43.us-west-1.compute.internal\" not found" node="ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:02.945399 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.945375    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 21:14:02.947017 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.946996    2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 21:14:02.963386 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.963358    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/104317fa8cfa5db2df29de73ade6b849-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal\" (UID: \"104317fa8cfa5db2df29de73ade6b849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:02.963454 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.963392    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/104317fa8cfa5db2df29de73ade6b849-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal\" (UID: \"104317fa8cfa5db2df29de73ade6b849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:02.963454 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.963411    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8253f9611b0fc9c6ff64af8baa405814-config\") pod \"kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal\" (UID: \"8253f9611b0fc9c6ff64af8baa405814\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:02.963515 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.963457    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/104317fa8cfa5db2df29de73ade6b849-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal\" (UID: \"104317fa8cfa5db2df29de73ade6b849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:02.963515 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.963464    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/104317fa8cfa5db2df29de73ade6b849-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal\" (UID: \"104317fa8cfa5db2df29de73ade6b849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:02.963579 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:02.963514    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8253f9611b0fc9c6ff64af8baa405814-config\") pod \"kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal\" (UID: \"8253f9611b0fc9c6ff64af8baa405814\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:03.043890 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.043848    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 21:14:03.045116 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.045093    2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 21:14:03.045648 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.045631    2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:03.045688 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:03.045660    2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-43.us-west-1.compute.internal\": node \"ip-10-0-128-43.us-west-1.compute.internal\" not found"
Nov 03 21:14:03.061116 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.061092    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:03.163964 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.163889    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 21:14:03.164103 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.164012    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:03.188158 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.188130    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:03.190305 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.190280    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:03.261441 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.261418    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 21:14:03.588134 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.588103    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 21:14:03.588714 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.588256    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 21:14:03.588714 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.588287    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 21:14:03.588714 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.588308    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 21:14:03.588714 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:03.588269    2574 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a272bc74d03104c58bb06e21328e27dc-41a6df2b1383438f.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-128-43.us-west-1.compute.internal.18e4a56dd10f953c\": read tcp 10.0.128.43:33746->16.58.154.219:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-128-43.us-west-1.compute.internal.18e4a56dd10f953c  default   4839 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-43.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-43.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-43.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 21:14:02 +0000 UTC,LastTimestamp:2026-11-03 21:14:02.838899174 +0000 UTC m=+0.580901198,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-43.us-west-1.compute.internal,}"
Nov 03 21:14:03.632289 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.632245    2574 apiserver.go:52] "Watching apiserver"
Nov 03 21:14:03.659946 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.659914    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 21:14:03.874773 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.874689    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 21:09:04 +0000 UTC" deadline="2028-07-28 13:14:52.149717566 +0000 UTC"
Nov 03 21:14:03.874773 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.874719    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15184h0m48.275001673s"
Nov 03 21:14:03.900197 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.900170    2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 21:14:03.900489 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.900474    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-67blc","openshift-network-diagnostics/network-check-target-8w5st","openshift-ovn-kubernetes/ovnkube-node-fvkn6","kube-system/global-pull-secret-syncer-v5vrf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr","openshift-multus/network-metrics-daemon-6xvgt","openshift-dns/node-resolver-554sk","openshift-network-operator/iptables-alerter-km5mx","kube-system/konnectivity-agent-cccr5","openshift-cluster-node-tuning-operator/tuned-6zndc","openshift-image-registry/node-ca-z5s4n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-f2pgd","kube-system/kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal"]
Nov 03 21:14:03.902934 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.902917    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-67blc"
Nov 03 21:14:03.905061 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.905041    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:03.905166 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:03.905117    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8w5st" podUID="784231bb-189f-472c-83ed-6ced092f10a1"
Nov 03 21:14:03.907266 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.907252    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:03.909298 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.909281    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:03.909369 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:03.909337    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5vrf" podUID="96fee653-58b4-46bf-9ed5-0d3c5a93e8a8"
Nov 03 21:14:03.911343 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.911328    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:03.913403 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.913387    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:03.913465 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:03.913440    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xvgt" podUID="4b84ba29-a116-48ef-8d9f-d81ceb4b3359"
Nov 03 21:14:03.915425 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.915412    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-554sk"
Nov 03 21:14:03.917543 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.917530    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-km5mx"
Nov 03 21:14:03.919770 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.919756    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cccr5"
Nov 03 21:14:03.922109 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.922089    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:03.924423 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.924403    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z5s4n"
Nov 03 21:14:03.926679 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.926662    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:03.957534 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.957508    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 21:14:03.957655 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.957623    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 21:14:03.957711 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.957668    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 21:14:03.957760 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.957748    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 21:14:03.957803 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.957749    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4f9ng\""
Nov 03 21:14:03.960002 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.959979    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 21:14:03.960110 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.960064    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 21:14:03.960110 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.960064    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 21:14:03.960216 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.960140    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 21:14:03.960216 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.960173    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 21:14:03.960316 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.960283    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 21:14:03.960357 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.960319    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vbwk2\""
Nov 03 21:14:03.961993 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.961976    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 21:14:03.963448 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.963433    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 21:14:03.963877 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.963844    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zzfft\""
Nov 03 21:14:03.963962 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.963912    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 21:14:03.963962 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.963924    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 21:14:03.967506 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.967490    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 21:14:03.967940 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.967924    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 21:14:03.968805 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.968787    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-sys-fs\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:03.968885 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.968814    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c3560fa9-2666-4e0d-854c-3c126489c61d-host\") pod \"node-ca-z5s4n\" (UID: \"c3560fa9-2666-4e0d-854c-3c126489c61d\") " pod="openshift-image-registry/node-ca-z5s4n"
Nov 03 21:14:03.968885 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.968829    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-tuned\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:03.968885 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.968845    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abf1a25c-1da1-449d-90c0-35d6fc406647-cni-binary-copy\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:03.969008 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.968881    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/abf1a25c-1da1-449d-90c0-35d6fc406647-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:03.969008 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.968909    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9046c34b-bb8f-4397-8b9d-7e2095add002-agent-certs\") pod \"konnectivity-agent-cccr5\" (UID: \"9046c34b-bb8f-4397-8b9d-7e2095add002\") " pod="kube-system/konnectivity-agent-cccr5"
Nov 03 21:14:03.969008 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.968928    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6fs8\" (UniqueName: \"kubernetes.io/projected/c3560fa9-2666-4e0d-854c-3c126489c61d-kube-api-access-l6fs8\") pod \"node-ca-z5s4n\" (UID: \"c3560fa9-2666-4e0d-854c-3c126489c61d\") " pod="openshift-image-registry/node-ca-z5s4n"
Nov 03 21:14:03.969008 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.968942    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-systemd\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:03.969008 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.968955    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-cnibin\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:03.969008 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.968969    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7qxh\" (UniqueName: \"kubernetes.io/projected/f63c6326-4f32-4fa3-925a-83743ef7ff39-kube-api-access-q7qxh\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:03.969008 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.968987    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e703c9a2-a51a-4de6-b30c-6cc2aa810e57-host-slash\") pod \"iptables-alerter-km5mx\" (UID: \"e703c9a2-a51a-4de6-b30c-6cc2aa810e57\") " pod="openshift-network-operator/iptables-alerter-km5mx"
Nov 03 21:14:03.969008 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969007    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-systemd-units\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:03.969334 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969022    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-system-cni-dir\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:03.969334 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969036    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-var-lib-kubelet\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:03.969334 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969068    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d34868b-605a-4bfb-81a3-432e0db0ce6a-tmp-dir\") pod \"node-resolver-554sk\" (UID: \"3d34868b-605a-4bfb-81a3-432e0db0ce6a\") " pod="openshift-dns/node-resolver-554sk"
Nov 03 21:14:03.969334 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969100    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-kubelet\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:03.969334 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969128    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-modprobe-d\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:03.969334 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969143    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-log-socket\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:03.969334 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969159    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a295913f-3811-47d6-9e85-a76182e45649-env-overrides\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:03.969334 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969198    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42d4m\" (UniqueName: \"kubernetes.io/projected/a295913f-3811-47d6-9e85-a76182e45649-kube-api-access-42d4m\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:03.969334 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969236    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-node-log\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:03.969334 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969259    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-cni-netd\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:03.969334 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969277    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a295913f-3811-47d6-9e85-a76182e45649-ovn-node-metrics-cert\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:03.969334 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969293    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-run\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:03.969334 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969308    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-lib-modules\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:03.969334 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969321    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-var-lib-kubelet\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:03.969334 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969336    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-run-netns\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:03.969728 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969361    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-run-netns\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:03.969728 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969378    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-kubernetes\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:03.969728 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969396    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-host\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:03.969728 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969418    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abf1a25c-1da1-449d-90c0-35d6fc406647-system-cni-dir\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:03.969728 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969435    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/abf1a25c-1da1-449d-90c0-35d6fc406647-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:03.969728 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969448    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-var-lib-cni-multus\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:03.969728 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969463    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-run-multus-certs\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:03.969728 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969486    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-socket-dir\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:03.969728 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969504    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:03.969728 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969520    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3d34868b-605a-4bfb-81a3-432e0db0ce6a-hosts-file\") pod \"node-resolver-554sk\" (UID: \"3d34868b-605a-4bfb-81a3-432e0db0ce6a\") " pod="openshift-dns/node-resolver-554sk"
Nov 03 21:14:03.969728 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969543    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a295913f-3811-47d6-9e85-a76182e45649-ovnkube-config\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:03.969728 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969559    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-sysctl-d\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:03.969728 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969572    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-sysctl-conf\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:03.969728 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969585    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f63c6326-4f32-4fa3-925a-83743ef7ff39-cni-binary-copy\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:03.969728 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969607    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4vrx\" (UniqueName: \"kubernetes.io/projected/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-kube-api-access-m4vrx\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:03.969728 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969638    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e703c9a2-a51a-4de6-b30c-6cc2aa810e57-iptables-alerter-script\") pod \"iptables-alerter-km5mx\" (UID: \"e703c9a2-a51a-4de6-b30c-6cc2aa810e57\") " pod="openshift-network-operator/iptables-alerter-km5mx"
Nov 03 21:14:03.970238 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969656    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9046c34b-bb8f-4397-8b9d-7e2095add002-konnectivity-ca\") pod \"konnectivity-agent-cccr5\" (UID: \"9046c34b-bb8f-4397-8b9d-7e2095add002\") " pod="kube-system/konnectivity-agent-cccr5"
Nov 03 21:14:03.970238 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969674    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abf1a25c-1da1-449d-90c0-35d6fc406647-cnibin\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:03.970238 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969689    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-registration-dir\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:03.970238 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969713    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv52h\" (UniqueName: \"kubernetes.io/projected/abf1a25c-1da1-449d-90c0-35d6fc406647-kube-api-access-tv52h\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:03.970238 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969731    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-multus-cni-dir\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:03.970238 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969772    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-original-pull-secret\") pod \"global-pull-secret-syncer-v5vrf\" (UID: \"96fee653-58b4-46bf-9ed5-0d3c5a93e8a8\") " pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:03.970238 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969797    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-run-systemd\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:03.970238 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969827    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-run-openvswitch\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:03.970238 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969909    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-run-ovn-kubernetes\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:03.970238 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969951    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:03.970238 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969971    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs\") pod \"network-metrics-daemon-6xvgt\" (UID: \"4b84ba29-a116-48ef-8d9f-d81ceb4b3359\") " pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:03.970238 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.969998    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-sys\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:03.970238 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970023    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23c56cf7-2a99-4df8-841a-d84b6a9a3822-tmp\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:03.970238 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970044    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt2rs\" (UniqueName: \"kubernetes.io/projected/23c56cf7-2a99-4df8-841a-d84b6a9a3822-kube-api-access-kt2rs\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:03.970238 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970068    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-var-lib-openvswitch\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:03.970238 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970093    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxp6s\" (UniqueName: \"kubernetes.io/projected/3d34868b-605a-4bfb-81a3-432e0db0ce6a-kube-api-access-wxp6s\") pod \"node-resolver-554sk\" (UID: \"3d34868b-605a-4bfb-81a3-432e0db0ce6a\") " pod="openshift-dns/node-resolver-554sk"
Nov 03 21:14:03.970733 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970111    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:03.970733 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970124    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-multus-conf-dir\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:03.970733 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970137    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-etc-kubernetes\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:03.970733 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970153    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c3560fa9-2666-4e0d-854c-3c126489c61d-serviceca\") pod \"node-ca-z5s4n\" (UID: \"c3560fa9-2666-4e0d-854c-3c126489c61d\") " pod="openshift-image-registry/node-ca-z5s4n"
Nov 03 21:14:03.970733 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970175    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-slash\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:03.970733 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970194    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abf1a25c-1da1-449d-90c0-35d6fc406647-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:03.970733 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970210    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-multus-socket-dir-parent\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:03.970733 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970226    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-run-k8s-cni-cncf-io\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:03.970733 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970245    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f63c6326-4f32-4fa3-925a-83743ef7ff39-multus-daemon-config\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:03.970733 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970279    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhskl\" (UniqueName: \"kubernetes.io/projected/e703c9a2-a51a-4de6-b30c-6cc2aa810e57-kube-api-access-dhskl\") pod \"iptables-alerter-km5mx\" (UID: \"e703c9a2-a51a-4de6-b30c-6cc2aa810e57\") " pod="openshift-network-operator/iptables-alerter-km5mx"
Nov 03 21:14:03.970733 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970305    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-etc-openvswitch\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:03.970733 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970320    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-run-ovn\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:03.970733 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970343    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p2xx\" (UniqueName: \"kubernetes.io/projected/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-kube-api-access-2p2xx\") pod \"network-metrics-daemon-6xvgt\" (UID: \"4b84ba29-a116-48ef-8d9f-d81ceb4b3359\") " pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:03.970733 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970370    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-dbus\") pod \"global-pull-secret-syncer-v5vrf\" (UID: \"96fee653-58b4-46bf-9ed5-0d3c5a93e8a8\") " pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:03.970733 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970395    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-device-dir\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:03.970733 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970413    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-cni-bin\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:03.971185 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970433    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-sysconfig\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:03.971185 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970479    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-kubelet-config\") pod \"global-pull-secret-syncer-v5vrf\" (UID: \"96fee653-58b4-46bf-9ed5-0d3c5a93e8a8\") " pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:03.971185 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970499    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb9kd\" (UniqueName: \"kubernetes.io/projected/784231bb-189f-472c-83ed-6ced092f10a1-kube-api-access-nb9kd\") pod \"network-check-target-8w5st\" (UID: \"784231bb-189f-472c-83ed-6ced092f10a1\") " pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:03.971185 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970513    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a295913f-3811-47d6-9e85-a76182e45649-ovnkube-script-lib\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:03.971185 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970527    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-var-lib-cni-bin\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:03.971185 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:03.970540    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-hostroot\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.003706 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.003686    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cldzt\""
Nov 03 21:14:04.003795 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.003755    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 21:14:04.005406 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.005392    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 21:14:04.007769 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.007753    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-trdz7\""
Nov 03 21:14:04.008033 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.008021    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 21:14:04.008486 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.008472    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lzq4j\""
Nov 03 21:14:04.008533 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.008499    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 21:14:04.010711 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.010698    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zbt6z\""
Nov 03 21:14:04.010977 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.010964    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 21:14:04.010977 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.010974    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 21:14:04.011093 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.011023    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 21:14:04.011147 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.011106    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 21:14:04.011196 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.011148    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w4rzx\""
Nov 03 21:14:04.011196 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.011163    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 21:14:04.011196 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.011184    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 21:14:04.011196 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.011191    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 21:14:04.011331 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.011185    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jxfhg\""
Nov 03 21:14:04.011331 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.011205    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 21:14:04.011331 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.011286    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 21:14:04.011437 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.011332    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cz4gm"
Nov 03 21:14:04.071609 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.071579    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-registration-dir\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:04.071733 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.071615    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv52h\" (UniqueName: \"kubernetes.io/projected/abf1a25c-1da1-449d-90c0-35d6fc406647-kube-api-access-tv52h\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:04.071733 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.071642    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-multus-cni-dir\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.071733 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.071666    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-original-pull-secret\") pod \"global-pull-secret-syncer-v5vrf\" (UID: \"96fee653-58b4-46bf-9ed5-0d3c5a93e8a8\") " pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:04.071733 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.071689    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-run-systemd\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.071733 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.071712    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-run-openvswitch\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.071733 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.071687    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-registration-dir\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:04.071976 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.071736    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-run-ovn-kubernetes\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.071976 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.071745    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-run-systemd\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.071976 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.071772    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-run-ovn-kubernetes\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.071976 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.071792    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-run-openvswitch\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.071976 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.071793    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-multus-cni-dir\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.071976 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:04.071816    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:04.071976 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.071851    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.071976 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:04.071901    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-original-pull-secret podName:96fee653-58b4-46bf-9ed5-0d3c5a93e8a8 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:04.571874074 +0000 UTC m=+2.313876095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-original-pull-secret") pod "global-pull-secret-syncer-v5vrf" (UID: "96fee653-58b4-46bf-9ed5-0d3c5a93e8a8") : object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:04.071976 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.071903    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.071976 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.071931    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs\") pod \"network-metrics-daemon-6xvgt\" (UID: \"4b84ba29-a116-48ef-8d9f-d81ceb4b3359\") " pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:04.072322 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072132    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-sys\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.072363 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.071961    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-sys\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.072401 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072373    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23c56cf7-2a99-4df8-841a-d84b6a9a3822-tmp\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.072401 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072390    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt2rs\" (UniqueName: \"kubernetes.io/projected/23c56cf7-2a99-4df8-841a-d84b6a9a3822-kube-api-access-kt2rs\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.072482 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072404    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-var-lib-openvswitch\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.072482 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072420    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxp6s\" (UniqueName: \"kubernetes.io/projected/3d34868b-605a-4bfb-81a3-432e0db0ce6a-kube-api-access-wxp6s\") pod \"node-resolver-554sk\" (UID: \"3d34868b-605a-4bfb-81a3-432e0db0ce6a\") " pod="openshift-dns/node-resolver-554sk"
Nov 03 21:14:04.072482 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:04.072433    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:04.072482 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:04.072476    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs podName:4b84ba29-a116-48ef-8d9f-d81ceb4b3359 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:04.572461555 +0000 UTC m=+2.314463564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs") pod "network-metrics-daemon-6xvgt" (UID: "4b84ba29-a116-48ef-8d9f-d81ceb4b3359") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:04.072482 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072436    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:04.072708 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072482    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:04.072708 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072510    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-multus-conf-dir\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.072708 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072528    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-etc-kubernetes\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.072708 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072534    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-var-lib-openvswitch\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.072708 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072546    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c3560fa9-2666-4e0d-854c-3c126489c61d-serviceca\") pod \"node-ca-z5s4n\" (UID: \"c3560fa9-2666-4e0d-854c-3c126489c61d\") " pod="openshift-image-registry/node-ca-z5s4n"
Nov 03 21:14:04.072708 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072570    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-multus-conf-dir\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.072708 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072598    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-slash\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.072708 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072629    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abf1a25c-1da1-449d-90c0-35d6fc406647-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:04.072708 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072600    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-etc-kubernetes\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.072708 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072655    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-multus-socket-dir-parent\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.072708 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072646    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-slash\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.072708 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072692    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-run-k8s-cni-cncf-io\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.072708 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072698    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-multus-socket-dir-parent\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.073295 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072730    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f63c6326-4f32-4fa3-925a-83743ef7ff39-multus-daemon-config\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.073295 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072734    2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 21:14:04.073295 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072757    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhskl\" (UniqueName: \"kubernetes.io/projected/e703c9a2-a51a-4de6-b30c-6cc2aa810e57-kube-api-access-dhskl\") pod \"iptables-alerter-km5mx\" (UID: \"e703c9a2-a51a-4de6-b30c-6cc2aa810e57\") " pod="openshift-network-operator/iptables-alerter-km5mx"
Nov 03 21:14:04.073295 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072762    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-run-k8s-cni-cncf-io\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.073295 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072780    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-etc-openvswitch\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.073295 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072787    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abf1a25c-1da1-449d-90c0-35d6fc406647-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:04.073295 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072804    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-run-ovn\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.073295 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072820    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-etc-openvswitch\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.073295 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072825    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p2xx\" (UniqueName: \"kubernetes.io/projected/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-kube-api-access-2p2xx\") pod \"network-metrics-daemon-6xvgt\" (UID: \"4b84ba29-a116-48ef-8d9f-d81ceb4b3359\") " pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:04.073295 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072896    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-run-ovn\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.073295 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072940    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-dbus\") pod \"global-pull-secret-syncer-v5vrf\" (UID: \"96fee653-58b4-46bf-9ed5-0d3c5a93e8a8\") " pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:04.073295 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072988    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-device-dir\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:04.073295 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.072995    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c3560fa9-2666-4e0d-854c-3c126489c61d-serviceca\") pod \"node-ca-z5s4n\" (UID: \"c3560fa9-2666-4e0d-854c-3c126489c61d\") " pod="openshift-image-registry/node-ca-z5s4n"
Nov 03 21:14:04.073295 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073004    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-dbus\") pod \"global-pull-secret-syncer-v5vrf\" (UID: \"96fee653-58b4-46bf-9ed5-0d3c5a93e8a8\") " pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:04.073295 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073012    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-cni-bin\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.073295 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073035    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-sysconfig\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.073295 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073047    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-device-dir\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:04.073295 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073062    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-kubelet-config\") pod \"global-pull-secret-syncer-v5vrf\" (UID: \"96fee653-58b4-46bf-9ed5-0d3c5a93e8a8\") " pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:04.074134 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073068    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-sysconfig\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.074134 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073069    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-cni-bin\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.074134 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073084    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb9kd\" (UniqueName: \"kubernetes.io/projected/784231bb-189f-472c-83ed-6ced092f10a1-kube-api-access-nb9kd\") pod \"network-check-target-8w5st\" (UID: \"784231bb-189f-472c-83ed-6ced092f10a1\") " pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:04.074134 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073107    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-kubelet-config\") pod \"global-pull-secret-syncer-v5vrf\" (UID: \"96fee653-58b4-46bf-9ed5-0d3c5a93e8a8\") " pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:04.074134 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073106    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a295913f-3811-47d6-9e85-a76182e45649-ovnkube-script-lib\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.074134 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073137    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-var-lib-cni-bin\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.074134 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073159    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-hostroot\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.074134 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073180    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-sys-fs\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:04.074134 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073200    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c3560fa9-2666-4e0d-854c-3c126489c61d-host\") pod \"node-ca-z5s4n\" (UID: \"c3560fa9-2666-4e0d-854c-3c126489c61d\") " pod="openshift-image-registry/node-ca-z5s4n"
Nov 03 21:14:04.074134 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073213    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-hostroot\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.074134 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073243    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-var-lib-cni-bin\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.074134 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073249    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-sys-fs\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:04.074134 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073241    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c3560fa9-2666-4e0d-854c-3c126489c61d-host\") pod \"node-ca-z5s4n\" (UID: \"c3560fa9-2666-4e0d-854c-3c126489c61d\") " pod="openshift-image-registry/node-ca-z5s4n"
Nov 03 21:14:04.074134 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073217    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-tuned\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.074134 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073293    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abf1a25c-1da1-449d-90c0-35d6fc406647-cni-binary-copy\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:04.074134 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073312    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/abf1a25c-1da1-449d-90c0-35d6fc406647-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:04.074134 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073326    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f63c6326-4f32-4fa3-925a-83743ef7ff39-multus-daemon-config\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.074134 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073336    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9046c34b-bb8f-4397-8b9d-7e2095add002-agent-certs\") pod \"konnectivity-agent-cccr5\" (UID: \"9046c34b-bb8f-4397-8b9d-7e2095add002\") " pod="kube-system/konnectivity-agent-cccr5"
Nov 03 21:14:04.074837 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073467    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6fs8\" (UniqueName: \"kubernetes.io/projected/c3560fa9-2666-4e0d-854c-3c126489c61d-kube-api-access-l6fs8\") pod \"node-ca-z5s4n\" (UID: \"c3560fa9-2666-4e0d-854c-3c126489c61d\") " pod="openshift-image-registry/node-ca-z5s4n"
Nov 03 21:14:04.074837 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073500    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-systemd\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.074837 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073526    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-cnibin\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.074837 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073552    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7qxh\" (UniqueName: \"kubernetes.io/projected/f63c6326-4f32-4fa3-925a-83743ef7ff39-kube-api-access-q7qxh\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.074837 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073575    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e703c9a2-a51a-4de6-b30c-6cc2aa810e57-host-slash\") pod \"iptables-alerter-km5mx\" (UID: \"e703c9a2-a51a-4de6-b30c-6cc2aa810e57\") " pod="openshift-network-operator/iptables-alerter-km5mx"
Nov 03 21:14:04.074837 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073601    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-systemd-units\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.074837 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073623    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-system-cni-dir\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.074837 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073649    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-var-lib-kubelet\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.074837 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073672    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d34868b-605a-4bfb-81a3-432e0db0ce6a-tmp-dir\") pod \"node-resolver-554sk\" (UID: \"3d34868b-605a-4bfb-81a3-432e0db0ce6a\") " pod="openshift-dns/node-resolver-554sk"
Nov 03 21:14:04.074837 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073673    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a295913f-3811-47d6-9e85-a76182e45649-ovnkube-script-lib\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.074837 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073715    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e703c9a2-a51a-4de6-b30c-6cc2aa810e57-host-slash\") pod \"iptables-alerter-km5mx\" (UID: \"e703c9a2-a51a-4de6-b30c-6cc2aa810e57\") " pod="openshift-network-operator/iptables-alerter-km5mx"
Nov 03 21:14:04.074837 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073698    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-kubelet\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.074837 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073750    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-modprobe-d\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.074837 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073749    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-systemd\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.074837 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073750    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-systemd-units\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.074837 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073784    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-cnibin\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.074837 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073797    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-var-lib-kubelet\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.074837 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073804    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-system-cni-dir\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.075578 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073839    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-kubelet\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.075578 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073897    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-log-socket\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.075578 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073924    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-modprobe-d\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.075578 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073932    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a295913f-3811-47d6-9e85-a76182e45649-env-overrides\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.075578 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073960    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42d4m\" (UniqueName: \"kubernetes.io/projected/a295913f-3811-47d6-9e85-a76182e45649-kube-api-access-42d4m\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.075578 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.073993    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-log-socket\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.075578 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074039    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-node-log\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.075578 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074071    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-cni-netd\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.075578 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074121    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a295913f-3811-47d6-9e85-a76182e45649-ovn-node-metrics-cert\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.075578 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074143    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-run\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.075578 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074165    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-lib-modules\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.075578 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074370    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-node-log\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.075578 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074409    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-var-lib-kubelet\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.075578 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074448    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-cni-netd\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.075578 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074371    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-var-lib-kubelet\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.075578 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074524    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abf1a25c-1da1-449d-90c0-35d6fc406647-cni-binary-copy\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:04.075578 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074532    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-run-netns\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.075578 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074602    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-run-netns\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.076379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074613    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-lib-modules\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.076379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074641    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-run-netns\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.076379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074657    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-kubernetes\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.076379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074666    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-run-netns\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.076379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074660    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-run\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.076379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074706    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-kubernetes\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.076379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074708    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-host\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.076379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074722    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d34868b-605a-4bfb-81a3-432e0db0ce6a-tmp-dir\") pod \"node-resolver-554sk\" (UID: \"3d34868b-605a-4bfb-81a3-432e0db0ce6a\") " pod="openshift-dns/node-resolver-554sk"
Nov 03 21:14:04.076379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074742    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-host\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.076379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074771    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abf1a25c-1da1-449d-90c0-35d6fc406647-system-cni-dir\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:04.076379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074822    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/abf1a25c-1da1-449d-90c0-35d6fc406647-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:04.076379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074822    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abf1a25c-1da1-449d-90c0-35d6fc406647-system-cni-dir\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:04.076379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074878    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-var-lib-cni-multus\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.076379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074909    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-run-multus-certs\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.076379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074917    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-var-lib-cni-multus\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.076379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074969    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-socket-dir\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:04.076379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.074996    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:04.076379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.075032    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3d34868b-605a-4bfb-81a3-432e0db0ce6a-hosts-file\") pod \"node-resolver-554sk\" (UID: \"3d34868b-605a-4bfb-81a3-432e0db0ce6a\") " pod="openshift-dns/node-resolver-554sk"
Nov 03 21:14:04.077171 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.075058    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-run-multus-certs\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.077171 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.075081    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a295913f-3811-47d6-9e85-a76182e45649-ovnkube-config\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.077171 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.075106    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-sysctl-d\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.077171 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.075142    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/abf1a25c-1da1-449d-90c0-35d6fc406647-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:04.077171 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.075156    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:04.077171 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.075157    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-sysctl-conf\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.077171 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.075211    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f63c6326-4f32-4fa3-925a-83743ef7ff39-cni-binary-copy\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.077171 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.075216    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3d34868b-605a-4bfb-81a3-432e0db0ce6a-hosts-file\") pod \"node-resolver-554sk\" (UID: \"3d34868b-605a-4bfb-81a3-432e0db0ce6a\") " pod="openshift-dns/node-resolver-554sk"
Nov 03 21:14:04.077171 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.075284    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/abf1a25c-1da1-449d-90c0-35d6fc406647-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:04.077171 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.075288    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-sysctl-conf\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.077171 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.075310    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-socket-dir\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:04.077171 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.075339    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4vrx\" (UniqueName: \"kubernetes.io/projected/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-kube-api-access-m4vrx\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:04.077171 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.075364    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e703c9a2-a51a-4de6-b30c-6cc2aa810e57-iptables-alerter-script\") pod \"iptables-alerter-km5mx\" (UID: \"e703c9a2-a51a-4de6-b30c-6cc2aa810e57\") " pod="openshift-network-operator/iptables-alerter-km5mx"
Nov 03 21:14:04.077171 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.075376    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-sysctl-d\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.077171 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.075387    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9046c34b-bb8f-4397-8b9d-7e2095add002-konnectivity-ca\") pod \"konnectivity-agent-cccr5\" (UID: \"9046c34b-bb8f-4397-8b9d-7e2095add002\") " pod="kube-system/konnectivity-agent-cccr5"
Nov 03 21:14:04.077171 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.075410    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abf1a25c-1da1-449d-90c0-35d6fc406647-cnibin\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:04.077171 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.076204    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f63c6326-4f32-4fa3-925a-83743ef7ff39-cni-binary-copy\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.077906 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.076411    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e703c9a2-a51a-4de6-b30c-6cc2aa810e57-iptables-alerter-script\") pod \"iptables-alerter-km5mx\" (UID: \"e703c9a2-a51a-4de6-b30c-6cc2aa810e57\") " pod="openshift-network-operator/iptables-alerter-km5mx"
Nov 03 21:14:04.077906 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.076716    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9046c34b-bb8f-4397-8b9d-7e2095add002-konnectivity-ca\") pod \"konnectivity-agent-cccr5\" (UID: \"9046c34b-bb8f-4397-8b9d-7e2095add002\") " pod="kube-system/konnectivity-agent-cccr5"
Nov 03 21:14:04.077906 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.076771    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abf1a25c-1da1-449d-90c0-35d6fc406647-cnibin\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:04.077906 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.077669    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a295913f-3811-47d6-9e85-a76182e45649-ovnkube-config\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.077906 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.077771    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23c56cf7-2a99-4df8-841a-d84b6a9a3822-tmp\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.077906 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.077885    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9046c34b-bb8f-4397-8b9d-7e2095add002-agent-certs\") pod \"konnectivity-agent-cccr5\" (UID: \"9046c34b-bb8f-4397-8b9d-7e2095add002\") " pod="kube-system/konnectivity-agent-cccr5"
Nov 03 21:14:04.080095 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.078291    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-tuned\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.080095 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.079877    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a295913f-3811-47d6-9e85-a76182e45649-ovn-node-metrics-cert\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.080673 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.080652    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a295913f-3811-47d6-9e85-a76182e45649-env-overrides\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.128291 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:04.128227    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 21:14:04.128291 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:04.128261    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 21:14:04.128291 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:04.128277    2574 projected.go:196] Error preparing data for projected volume kube-api-access-nb9kd for pod openshift-network-diagnostics/network-check-target-8w5st: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:04.128606 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:04.128359    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/784231bb-189f-472c-83ed-6ced092f10a1-kube-api-access-nb9kd podName:784231bb-189f-472c-83ed-6ced092f10a1 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:04.628337101 +0000 UTC m=+2.370339130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nb9kd" (UniqueName: "kubernetes.io/projected/784231bb-189f-472c-83ed-6ced092f10a1-kube-api-access-nb9kd") pod "network-check-target-8w5st" (UID: "784231bb-189f-472c-83ed-6ced092f10a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:04.130466 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.130437    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv52h\" (UniqueName: \"kubernetes.io/projected/abf1a25c-1da1-449d-90c0-35d6fc406647-kube-api-access-tv52h\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:04.130683 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.130663    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p2xx\" (UniqueName: \"kubernetes.io/projected/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-kube-api-access-2p2xx\") pod \"network-metrics-daemon-6xvgt\" (UID: \"4b84ba29-a116-48ef-8d9f-d81ceb4b3359\") " pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:04.130962 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.130940    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt2rs\" (UniqueName: \"kubernetes.io/projected/23c56cf7-2a99-4df8-841a-d84b6a9a3822-kube-api-access-kt2rs\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:04.131162 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.131140    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxp6s\" (UniqueName: \"kubernetes.io/projected/3d34868b-605a-4bfb-81a3-432e0db0ce6a-kube-api-access-wxp6s\") pod \"node-resolver-554sk\" (UID: \"3d34868b-605a-4bfb-81a3-432e0db0ce6a\") " pod="openshift-dns/node-resolver-554sk"
Nov 03 21:14:04.131562 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.131527    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42d4m\" (UniqueName: \"kubernetes.io/projected/a295913f-3811-47d6-9e85-a76182e45649-kube-api-access-42d4m\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:04.131658 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.131589    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhskl\" (UniqueName: \"kubernetes.io/projected/e703c9a2-a51a-4de6-b30c-6cc2aa810e57-kube-api-access-dhskl\") pod \"iptables-alerter-km5mx\" (UID: \"e703c9a2-a51a-4de6-b30c-6cc2aa810e57\") " pod="openshift-network-operator/iptables-alerter-km5mx"
Nov 03 21:14:04.131855 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.131836    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7qxh\" (UniqueName: \"kubernetes.io/projected/f63c6326-4f32-4fa3-925a-83743ef7ff39-kube-api-access-q7qxh\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:04.132013 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.131995    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6fs8\" (UniqueName: \"kubernetes.io/projected/c3560fa9-2666-4e0d-854c-3c126489c61d-kube-api-access-l6fs8\") pod \"node-ca-z5s4n\" (UID: \"c3560fa9-2666-4e0d-854c-3c126489c61d\") " pod="openshift-image-registry/node-ca-z5s4n"
Nov 03 21:14:04.136349 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.136325    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4vrx\" (UniqueName: \"kubernetes.io/projected/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-kube-api-access-m4vrx\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:04.228679 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:04.228651    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-67blc"
Nov 03 21:14:05.713048 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:05.713024    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:05.714558 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:05.714528    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104317fa8cfa5db2df29de73ade6b849.slice/crio-19c349dd462dcbcf83ae4a8b956193a1b9484ecd552fa9ed773e410c18d16d79 WatchSource:0}: Error finding container 19c349dd462dcbcf83ae4a8b956193a1b9484ecd552fa9ed773e410c18d16d79: Status 404 returned error can't find the container with id 19c349dd462dcbcf83ae4a8b956193a1b9484ecd552fa9ed773e410c18d16d79
Nov 03 21:14:05.714802 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:05.714777    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8253f9611b0fc9c6ff64af8baa405814.slice/crio-0a15829a8bd8ad5d05c335ab39dbdf0c9aa529ac4f64fcb36679332864076283 WatchSource:0}: Error finding container 0a15829a8bd8ad5d05c335ab39dbdf0c9aa529ac4f64fcb36679332864076283: Status 404 returned error can't find the container with id 0a15829a8bd8ad5d05c335ab39dbdf0c9aa529ac4f64fcb36679332864076283
Nov 03 21:14:05.718593 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:05.718571    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:05.720153 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:05.720133    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:14:05.723540 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:05.723515    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda295913f_3811_47d6_9e85_a76182e45649.slice/crio-654d663557e8f82d99dcdd5803bd5874617cadba42096308eff8d1f02824feab WatchSource:0}: Error finding container 654d663557e8f82d99dcdd5803bd5874617cadba42096308eff8d1f02824feab: Status 404 returned error can't find the container with id 654d663557e8f82d99dcdd5803bd5874617cadba42096308eff8d1f02824feab
Nov 03 21:14:05.728888 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:05.728869    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf83a41bd_5f2e_4c82_a357_ddb5be4be2f6.slice/crio-80027e2d8d189398ede870f89b1531f21dc99a0fdf39423eabc08a2f653f698a WatchSource:0}: Error finding container 80027e2d8d189398ede870f89b1531f21dc99a0fdf39423eabc08a2f653f698a: Status 404 returned error can't find the container with id 80027e2d8d189398ede870f89b1531f21dc99a0fdf39423eabc08a2f653f698a
Nov 03 21:14:05.729454 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:05.729439    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-554sk"
Nov 03 21:14:05.738227 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:05.738209    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d34868b_605a_4bfb_81a3_432e0db0ce6a.slice/crio-324ab33cc272046022d3836ad1f36a85253684b77101adc607fdacf44b143a2b WatchSource:0}: Error finding container 324ab33cc272046022d3836ad1f36a85253684b77101adc607fdacf44b143a2b: Status 404 returned error can't find the container with id 324ab33cc272046022d3836ad1f36a85253684b77101adc607fdacf44b143a2b
Nov 03 21:14:05.742230 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:05.742214    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-km5mx"
Nov 03 21:14:05.751018 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:05.751000    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode703c9a2_a51a_4de6_b30c_6cc2aa810e57.slice/crio-e140f3f9baecf10768662ff7b249a07618d4ac75cd3a24775af125ba8a66caff WatchSource:0}: Error finding container e140f3f9baecf10768662ff7b249a07618d4ac75cd3a24775af125ba8a66caff: Status 404 returned error can't find the container with id e140f3f9baecf10768662ff7b249a07618d4ac75cd3a24775af125ba8a66caff
Nov 03 21:14:05.759021 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:05.759006    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cccr5"
Nov 03 21:14:05.767519 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:05.767497    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9046c34b_bb8f_4397_8b9d_7e2095add002.slice/crio-097b094bd2ef653693302fbc8d2d34b595702dd18f0ece22727dec3df5532e2e WatchSource:0}: Error finding container 097b094bd2ef653693302fbc8d2d34b595702dd18f0ece22727dec3df5532e2e: Status 404 returned error can't find the container with id 097b094bd2ef653693302fbc8d2d34b595702dd18f0ece22727dec3df5532e2e
Nov 03 21:14:05.772520 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:05.772506    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:05.777093 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:05.777077    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z5s4n"
Nov 03 21:14:05.780898 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:05.780880    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c56cf7_2a99_4df8_841a_d84b6a9a3822.slice/crio-7f9a656db7b50918c3f769b7d92dcca722755fa4c253610c31e80826ff697143 WatchSource:0}: Error finding container 7f9a656db7b50918c3f769b7d92dcca722755fa4c253610c31e80826ff697143: Status 404 returned error can't find the container with id 7f9a656db7b50918c3f769b7d92dcca722755fa4c253610c31e80826ff697143
Nov 03 21:14:05.785323 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:05.785305    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3560fa9_2666_4e0d_854c_3c126489c61d.slice/crio-9a466dd4e385f3acf96c0d1cffe5a1abf0e239fe69b6417bbf62327a7b9a66cf WatchSource:0}: Error finding container 9a466dd4e385f3acf96c0d1cffe5a1abf0e239fe69b6417bbf62327a7b9a66cf: Status 404 returned error can't find the container with id 9a466dd4e385f3acf96c0d1cffe5a1abf0e239fe69b6417bbf62327a7b9a66cf
Nov 03 21:14:05.805335 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:05.805316    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:05.813840 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:05.813821    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf1a25c_1da1_449d_90c0_35d6fc406647.slice/crio-0f7c57ffd61c558ae200d75efe4615c846e43a8fbfc892cb5dec957acbfab481 WatchSource:0}: Error finding container 0f7c57ffd61c558ae200d75efe4615c846e43a8fbfc892cb5dec957acbfab481: Status 404 returned error can't find the container with id 0f7c57ffd61c558ae200d75efe4615c846e43a8fbfc892cb5dec957acbfab481
Nov 03 21:14:06.022129 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:06.022095    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-original-pull-secret\") pod \"global-pull-secret-syncer-v5vrf\" (UID: \"96fee653-58b4-46bf-9ed5-0d3c5a93e8a8\") " pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:06.022129 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:06.022127    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs\") pod \"network-metrics-daemon-6xvgt\" (UID: \"4b84ba29-a116-48ef-8d9f-d81ceb4b3359\") " pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:06.022354 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:06.022257    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:06.022354 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:06.022262    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:06.022354 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:06.022314    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs podName:4b84ba29-a116-48ef-8d9f-d81ceb4b3359 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:07.022297097 +0000 UTC m=+3.320923876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs") pod "network-metrics-daemon-6xvgt" (UID: "4b84ba29-a116-48ef-8d9f-d81ceb4b3359") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:06.022354 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:06.022328    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-original-pull-secret podName:96fee653-58b4-46bf-9ed5-0d3c5a93e8a8 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:07.022321658 +0000 UTC m=+3.320948415 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-original-pull-secret") pod "global-pull-secret-syncer-v5vrf" (UID: "96fee653-58b4-46bf-9ed5-0d3c5a93e8a8") : object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:06.110152 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:06.110120    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 21:14:06.122935 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:06.122910    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb9kd\" (UniqueName: \"kubernetes.io/projected/784231bb-189f-472c-83ed-6ced092f10a1-kube-api-access-nb9kd\") pod \"network-check-target-8w5st\" (UID: \"784231bb-189f-472c-83ed-6ced092f10a1\") " pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:06.123082 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:06.123067    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 21:14:06.123117 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:06.123088    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 21:14:06.123117 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:06.123098    2574 projected.go:196] Error preparing data for projected volume kube-api-access-nb9kd for pod openshift-network-diagnostics/network-check-target-8w5st: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:06.123184 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:06.123143    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/784231bb-189f-472c-83ed-6ced092f10a1-kube-api-access-nb9kd podName:784231bb-189f-472c-83ed-6ced092f10a1 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:07.123129883 +0000 UTC m=+3.421756658 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nb9kd" (UniqueName: "kubernetes.io/projected/784231bb-189f-472c-83ed-6ced092f10a1-kube-api-access-nb9kd") pod "network-check-target-8w5st" (UID: "784231bb-189f-472c-83ed-6ced092f10a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:06.191254 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:06.191188    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2pgd" event={"ID":"abf1a25c-1da1-449d-90c0-35d6fc406647","Type":"ContainerStarted","Data":"0f7c57ffd61c558ae200d75efe4615c846e43a8fbfc892cb5dec957acbfab481"}
Nov 03 21:14:06.192318 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:06.192295    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6zndc" event={"ID":"23c56cf7-2a99-4df8-841a-d84b6a9a3822","Type":"ContainerStarted","Data":"7f9a656db7b50918c3f769b7d92dcca722755fa4c253610c31e80826ff697143"}
Nov 03 21:14:06.193366 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:06.193348    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cccr5" event={"ID":"9046c34b-bb8f-4397-8b9d-7e2095add002","Type":"ContainerStarted","Data":"097b094bd2ef653693302fbc8d2d34b595702dd18f0ece22727dec3df5532e2e"}
Nov 03 21:14:06.194455 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:06.194438    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-km5mx" event={"ID":"e703c9a2-a51a-4de6-b30c-6cc2aa810e57","Type":"ContainerStarted","Data":"e140f3f9baecf10768662ff7b249a07618d4ac75cd3a24775af125ba8a66caff"}
Nov 03 21:14:06.195506 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:06.195485    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-554sk" event={"ID":"3d34868b-605a-4bfb-81a3-432e0db0ce6a","Type":"ContainerStarted","Data":"324ab33cc272046022d3836ad1f36a85253684b77101adc607fdacf44b143a2b"}
Nov 03 21:14:06.196459 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:06.196440    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr" event={"ID":"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6","Type":"ContainerStarted","Data":"80027e2d8d189398ede870f89b1531f21dc99a0fdf39423eabc08a2f653f698a"}
Nov 03 21:14:06.197470 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:06.197447    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal" event={"ID":"104317fa8cfa5db2df29de73ade6b849","Type":"ContainerStarted","Data":"19c349dd462dcbcf83ae4a8b956193a1b9484ecd552fa9ed773e410c18d16d79"}
Nov 03 21:14:06.198359 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:06.198343    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal" event={"ID":"8253f9611b0fc9c6ff64af8baa405814","Type":"ContainerStarted","Data":"0a15829a8bd8ad5d05c335ab39dbdf0c9aa529ac4f64fcb36679332864076283"}
Nov 03 21:14:06.199344 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:06.199317    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z5s4n" event={"ID":"c3560fa9-2666-4e0d-854c-3c126489c61d","Type":"ContainerStarted","Data":"9a466dd4e385f3acf96c0d1cffe5a1abf0e239fe69b6417bbf62327a7b9a66cf"}
Nov 03 21:14:06.200419 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:06.200401    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6" event={"ID":"a295913f-3811-47d6-9e85-a76182e45649","Type":"ContainerStarted","Data":"654d663557e8f82d99dcdd5803bd5874617cadba42096308eff8d1f02824feab"}
Nov 03 21:14:06.261637 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:06.261602    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 21:14:06.457341 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:06.457283    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 21:09:05 +0000 UTC" deadline="2028-08-19 22:44:30.9999406 +0000 UTC"
Nov 03 21:14:06.457341 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:06.457335    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15721h30m24.542608222s"
Nov 03 21:14:06.503688 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:06.503657    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf63c6326_4f32_4fa3_925a_83743ef7ff39.slice/crio-047235fabf11c01f841f93ff511e942d78c3561e32d9afb78b1ee150f1fcc226 WatchSource:0}: Error finding container 047235fabf11c01f841f93ff511e942d78c3561e32d9afb78b1ee150f1fcc226: Status 404 returned error can't find the container with id 047235fabf11c01f841f93ff511e942d78c3561e32d9afb78b1ee150f1fcc226
Nov 03 21:14:06.540995 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:06.540970    2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 21:14:07.030339 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:07.029880    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs\") pod \"network-metrics-daemon-6xvgt\" (UID: \"4b84ba29-a116-48ef-8d9f-d81ceb4b3359\") " pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:07.030339 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:07.029987    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-original-pull-secret\") pod \"global-pull-secret-syncer-v5vrf\" (UID: \"96fee653-58b4-46bf-9ed5-0d3c5a93e8a8\") " pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:07.030339 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:07.030110    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:07.030339 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:07.030187    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-original-pull-secret podName:96fee653-58b4-46bf-9ed5-0d3c5a93e8a8 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:09.030169435 +0000 UTC m=+5.328796212 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-original-pull-secret") pod "global-pull-secret-syncer-v5vrf" (UID: "96fee653-58b4-46bf-9ed5-0d3c5a93e8a8") : object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:07.030339 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:07.030269    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:07.030339 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:07.030303    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs podName:4b84ba29-a116-48ef-8d9f-d81ceb4b3359 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:09.030291676 +0000 UTC m=+5.328918448 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs") pod "network-metrics-daemon-6xvgt" (UID: "4b84ba29-a116-48ef-8d9f-d81ceb4b3359") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:07.130345 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:07.130298    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb9kd\" (UniqueName: \"kubernetes.io/projected/784231bb-189f-472c-83ed-6ced092f10a1-kube-api-access-nb9kd\") pod \"network-check-target-8w5st\" (UID: \"784231bb-189f-472c-83ed-6ced092f10a1\") " pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:07.130939 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:07.130540    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 21:14:07.130939 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:07.130560    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 21:14:07.130939 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:07.130574    2574 projected.go:196] Error preparing data for projected volume kube-api-access-nb9kd for pod openshift-network-diagnostics/network-check-target-8w5st: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:07.130939 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:07.130632    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/784231bb-189f-472c-83ed-6ced092f10a1-kube-api-access-nb9kd podName:784231bb-189f-472c-83ed-6ced092f10a1 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:09.130614394 +0000 UTC m=+5.429241155 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nb9kd" (UniqueName: "kubernetes.io/projected/784231bb-189f-472c-83ed-6ced092f10a1-kube-api-access-nb9kd") pod "network-check-target-8w5st" (UID: "784231bb-189f-472c-83ed-6ced092f10a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:07.187188 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:07.187142    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:07.187382 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:07.187336    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xvgt" podUID="4b84ba29-a116-48ef-8d9f-d81ceb4b3359"
Nov 03 21:14:07.187456 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:07.187426    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:07.187526 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:07.187499    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5vrf" podUID="96fee653-58b4-46bf-9ed5-0d3c5a93e8a8"
Nov 03 21:14:07.187579 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:07.187561    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:07.187635 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:07.187621    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8w5st" podUID="784231bb-189f-472c-83ed-6ced092f10a1"
Nov 03 21:14:07.225748 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:07.225692    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-67blc" event={"ID":"f63c6326-4f32-4fa3-925a-83743ef7ff39","Type":"ContainerStarted","Data":"047235fabf11c01f841f93ff511e942d78c3561e32d9afb78b1ee150f1fcc226"}
Nov 03 21:14:07.459606 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:07.458306    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 21:09:05 +0000 UTC" deadline="2028-07-13 02:23:41.883653278 +0000 UTC"
Nov 03 21:14:07.459606 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:07.458398    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14813h9m34.425260902s"
Nov 03 21:14:09.049005 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:09.048968    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-original-pull-secret\") pod \"global-pull-secret-syncer-v5vrf\" (UID: \"96fee653-58b4-46bf-9ed5-0d3c5a93e8a8\") " pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:09.049005 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:09.049011    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs\") pod \"network-metrics-daemon-6xvgt\" (UID: \"4b84ba29-a116-48ef-8d9f-d81ceb4b3359\") " pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:09.049581 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:09.049131    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:09.049581 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:09.049192    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs podName:4b84ba29-a116-48ef-8d9f-d81ceb4b3359 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:13.049175257 +0000 UTC m=+9.347802020 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs") pod "network-metrics-daemon-6xvgt" (UID: "4b84ba29-a116-48ef-8d9f-d81ceb4b3359") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:09.049581 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:09.049281    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:09.049581 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:09.049315    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-original-pull-secret podName:96fee653-58b4-46bf-9ed5-0d3c5a93e8a8 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:13.049304739 +0000 UTC m=+9.347931517 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-original-pull-secret") pod "global-pull-secret-syncer-v5vrf" (UID: "96fee653-58b4-46bf-9ed5-0d3c5a93e8a8") : object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:09.149918 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:09.149878    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb9kd\" (UniqueName: \"kubernetes.io/projected/784231bb-189f-472c-83ed-6ced092f10a1-kube-api-access-nb9kd\") pod \"network-check-target-8w5st\" (UID: \"784231bb-189f-472c-83ed-6ced092f10a1\") " pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:09.150215 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:09.150192    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 21:14:09.150356 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:09.150223    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 21:14:09.150356 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:09.150254    2574 projected.go:196] Error preparing data for projected volume kube-api-access-nb9kd for pod openshift-network-diagnostics/network-check-target-8w5st: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:09.150465 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:09.150370    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/784231bb-189f-472c-83ed-6ced092f10a1-kube-api-access-nb9kd podName:784231bb-189f-472c-83ed-6ced092f10a1 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:13.150326146 +0000 UTC m=+9.448963396 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nb9kd" (UniqueName: "kubernetes.io/projected/784231bb-189f-472c-83ed-6ced092f10a1-kube-api-access-nb9kd") pod "network-check-target-8w5st" (UID: "784231bb-189f-472c-83ed-6ced092f10a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:09.186914 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:09.186879    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:09.187080 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:09.187009    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5vrf" podUID="96fee653-58b4-46bf-9ed5-0d3c5a93e8a8"
Nov 03 21:14:09.187139 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:09.187082    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:09.187202 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:09.187154    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8w5st" podUID="784231bb-189f-472c-83ed-6ced092f10a1"
Nov 03 21:14:09.187314 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:09.187209    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:09.187314 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:09.187305    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xvgt" podUID="4b84ba29-a116-48ef-8d9f-d81ceb4b3359"
Nov 03 21:14:11.186685 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:11.186615    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:11.187118 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:11.186731    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xvgt" podUID="4b84ba29-a116-48ef-8d9f-d81ceb4b3359"
Nov 03 21:14:11.187118 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:11.186776    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:11.187118 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:11.186820    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5vrf" podUID="96fee653-58b4-46bf-9ed5-0d3c5a93e8a8"
Nov 03 21:14:11.187118 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:11.186864    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:11.187118 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:11.186900    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8w5st" podUID="784231bb-189f-472c-83ed-6ced092f10a1"
Nov 03 21:14:11.258855 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:11.258803    2574 generic.go:356] "Generic (PLEG): container finished" podID="abf1a25c-1da1-449d-90c0-35d6fc406647" containerID="3046d10cfc825df3046d4a961c19bf219066078ea1b9753c9457424084e2fc3e" exitCode=0
Nov 03 21:14:11.259256 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:11.259081    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2pgd" event={"ID":"abf1a25c-1da1-449d-90c0-35d6fc406647","Type":"ContainerDied","Data":"3046d10cfc825df3046d4a961c19bf219066078ea1b9753c9457424084e2fc3e"}
Nov 03 21:14:11.266071 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:11.265532    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cccr5" event={"ID":"9046c34b-bb8f-4397-8b9d-7e2095add002","Type":"ContainerStarted","Data":"553fcb9e9e9193f1356aedaf5403f9f654173bdf153b8c75eb4f40ec96db7e2f"}
Nov 03 21:14:11.271398 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:11.270773    2574 generic.go:356] "Generic (PLEG): container finished" podID="104317fa8cfa5db2df29de73ade6b849" containerID="616f6bda8df912c55f680ef7b4ba29e03af421160c74f93e5f090568611268aa" exitCode=0
Nov 03 21:14:11.271398 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:11.270847    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal" event={"ID":"104317fa8cfa5db2df29de73ade6b849","Type":"ContainerDied","Data":"616f6bda8df912c55f680ef7b4ba29e03af421160c74f93e5f090568611268aa"}
Nov 03 21:14:11.277069 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:11.276617    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal" event={"ID":"8253f9611b0fc9c6ff64af8baa405814","Type":"ContainerStarted","Data":"c2db52cc75ad5d65a686d94d85d6221bb9d797177fbd32be105d50d77e5d77be"}
Nov 03 21:14:11.658162 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:11.658099    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal" podStartSLOduration=7.658081925 podStartE2EDuration="7.658081925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:11.658003361 +0000 UTC m=+7.956630139" watchObservedRunningTime="2026-11-03 21:14:11.658081925 +0000 UTC m=+7.956708706"
Nov 03 21:14:11.772191 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:11.771691    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cccr5" podStartSLOduration=2.917562038 podStartE2EDuration="7.771678233s" totalImagesPullingTime="4.854116195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:11.771488457 +0000 UTC m=+8.070115239" watchObservedRunningTime="2026-11-03 21:14:11.771678233 +0000 UTC m=+8.070305013"
Nov 03 21:14:12.282373 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:12.282335    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal" event={"ID":"104317fa8cfa5db2df29de73ade6b849","Type":"ContainerStarted","Data":"4f6bdf9728a3f68fad69573e80101c16ce5d15295a368fb4a036da1cf93760f1"}
Nov 03 21:14:12.286257 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:12.285784    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z5s4n" event={"ID":"c3560fa9-2666-4e0d-854c-3c126489c61d","Type":"ContainerStarted","Data":"46dad12904a41a59a121d51f10a5dd022eb898e7e12a18d80619fcbf7ec1d0e6"}
Nov 03 21:14:12.398637 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:12.397871    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal" podStartSLOduration=8.397851053 podStartE2EDuration="8.397851053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:12.397459615 +0000 UTC m=+8.696086395" watchObservedRunningTime="2026-11-03 21:14:12.397851053 +0000 UTC m=+8.696477835"
Nov 03 21:14:13.079603 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:13.079561    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-original-pull-secret\") pod \"global-pull-secret-syncer-v5vrf\" (UID: \"96fee653-58b4-46bf-9ed5-0d3c5a93e8a8\") " pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:13.079775 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:13.079626    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs\") pod \"network-metrics-daemon-6xvgt\" (UID: \"4b84ba29-a116-48ef-8d9f-d81ceb4b3359\") " pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:13.080029 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:13.079815    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:13.080029 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:13.079882    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs podName:4b84ba29-a116-48ef-8d9f-d81ceb4b3359 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:21.079864721 +0000 UTC m=+17.378491493 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs") pod "network-metrics-daemon-6xvgt" (UID: "4b84ba29-a116-48ef-8d9f-d81ceb4b3359") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:13.080029 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:13.079950    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:13.080029 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:13.079988    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-original-pull-secret podName:96fee653-58b4-46bf-9ed5-0d3c5a93e8a8 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:21.079976235 +0000 UTC m=+17.378603007 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-original-pull-secret") pod "global-pull-secret-syncer-v5vrf" (UID: "96fee653-58b4-46bf-9ed5-0d3c5a93e8a8") : object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:13.180226 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:13.180186    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb9kd\" (UniqueName: \"kubernetes.io/projected/784231bb-189f-472c-83ed-6ced092f10a1-kube-api-access-nb9kd\") pod \"network-check-target-8w5st\" (UID: \"784231bb-189f-472c-83ed-6ced092f10a1\") " pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:13.180455 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:13.180422    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 21:14:13.180455 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:13.180448    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 21:14:13.180597 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:13.180461    2574 projected.go:196] Error preparing data for projected volume kube-api-access-nb9kd for pod openshift-network-diagnostics/network-check-target-8w5st: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:13.180597 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:13.180528    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/784231bb-189f-472c-83ed-6ced092f10a1-kube-api-access-nb9kd podName:784231bb-189f-472c-83ed-6ced092f10a1 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:21.180509376 +0000 UTC m=+17.479136151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nb9kd" (UniqueName: "kubernetes.io/projected/784231bb-189f-472c-83ed-6ced092f10a1-kube-api-access-nb9kd") pod "network-check-target-8w5st" (UID: "784231bb-189f-472c-83ed-6ced092f10a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:13.187519 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:13.187484    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:13.187675 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:13.187500    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:13.187675 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:13.187647    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5vrf" podUID="96fee653-58b4-46bf-9ed5-0d3c5a93e8a8"
Nov 03 21:14:13.187793 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:13.187676    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:13.187848 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:13.187813    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xvgt" podUID="4b84ba29-a116-48ef-8d9f-d81ceb4b3359"
Nov 03 21:14:13.187913 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:13.187893    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8w5st" podUID="784231bb-189f-472c-83ed-6ced092f10a1"
Nov 03 21:14:13.290042 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:13.290007    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr" event={"ID":"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6","Type":"ContainerStarted","Data":"0a2139bf002b164d77aa265450999e49c699558c4fbd7e3822915cf73a28c009"}
Nov 03 21:14:14.662784 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:14.662139    2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 21:14:15.186369 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:15.186191    2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T21:14:14.662159902Z","UUID":"d7a62731-417a-4512-a156-7489737227b2","Handler":null,"Name":"","Endpoint":""}
Nov 03 21:14:15.187388 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:15.187350    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:15.187388 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:15.187370    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:15.187537 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:15.187474    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5vrf" podUID="96fee653-58b4-46bf-9ed5-0d3c5a93e8a8"
Nov 03 21:14:15.187679 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:15.187648    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xvgt" podUID="4b84ba29-a116-48ef-8d9f-d81ceb4b3359"
Nov 03 21:14:15.187737 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:15.187699    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:15.187872 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:15.187846    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8w5st" podUID="784231bb-189f-472c-83ed-6ced092f10a1"
Nov 03 21:14:15.188370 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:15.188347    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 21:14:15.188445 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:15.188407    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 21:14:15.304401 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:15.304357    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-554sk" event={"ID":"3d34868b-605a-4bfb-81a3-432e0db0ce6a","Type":"ContainerStarted","Data":"67d19dd66523e7176a28e75043f2ce8996359ef33c659a8e8ba461104aa0673a"}
Nov 03 21:14:15.307060 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:15.307029    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr" event={"ID":"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6","Type":"ContainerStarted","Data":"39cf19355d2dc960ed5b20311085363a39e358e05116cb11c855ec4c7f345036"}
Nov 03 21:14:15.308739 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:15.308717    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-67blc" event={"ID":"f63c6326-4f32-4fa3-925a-83743ef7ff39","Type":"ContainerStarted","Data":"ff2cde9f6a450e577240dcb4c70691e5dee4db5ce9fd2b6c800aa9abe18fee44"}
Nov 03 21:14:15.419356 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:15.418869    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-554sk" podStartSLOduration=2.783714138 podStartE2EDuration="11.418851743s" totalImagesPullingTime="8.635137605s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:15.418328135 +0000 UTC m=+11.716954928" watchObservedRunningTime="2026-11-03 21:14:15.418851743 +0000 UTC m=+11.717478524"
Nov 03 21:14:15.419356 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:15.419136    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z5s4n" podStartSLOduration=5.365032715 podStartE2EDuration="11.419126048s" totalImagesPullingTime="6.054093333s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:12.511596829 +0000 UTC m=+8.810223610" watchObservedRunningTime="2026-11-03 21:14:15.419126048 +0000 UTC m=+11.717752832"
Nov 03 21:14:15.440020 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:15.439909    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cccr5"
Nov 03 21:14:15.441254 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:15.441213    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cccr5"
Nov 03 21:14:15.647849 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:15.647782    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-67blc" podStartSLOduration=3.2880392560000002 podStartE2EDuration="11.647763668s" totalImagesPullingTime="8.359724412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:15.533489096 +0000 UTC m=+11.832115871" watchObservedRunningTime="2026-11-03 21:14:15.647763668 +0000 UTC m=+11.946390484"
Nov 03 21:14:16.313970 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:16.313929    2574 generic.go:356] "Generic (PLEG): container finished" podID="abf1a25c-1da1-449d-90c0-35d6fc406647" containerID="272e72e384ca30edaa0952cbe619c2d5cdc9431d9926e09a61b62a5d9221045b" exitCode=0
Nov 03 21:14:16.314420 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:16.314026    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2pgd" event={"ID":"abf1a25c-1da1-449d-90c0-35d6fc406647","Type":"ContainerDied","Data":"272e72e384ca30edaa0952cbe619c2d5cdc9431d9926e09a61b62a5d9221045b"}
Nov 03 21:14:16.316821 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:16.316793    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-km5mx" event={"ID":"e703c9a2-a51a-4de6-b30c-6cc2aa810e57","Type":"ContainerStarted","Data":"29159e162e72c3725eff6dab0e17e688c19c93b3946a3b587c8d095e516b6d27"}
Nov 03 21:14:16.319865 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:16.319817    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr" event={"ID":"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6","Type":"ContainerStarted","Data":"c9e7c200b101b8e3574f6d52837acee176ed2b64564f6acb1c0fed07d7aad6f7"}
Nov 03 21:14:16.555422 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:16.555371    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-km5mx" podStartSLOduration=3.8122699559999997 podStartE2EDuration="12.555355073s" totalImagesPullingTime="8.743085117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:16.546608586 +0000 UTC m=+12.845235367" watchObservedRunningTime="2026-11-03 21:14:16.555355073 +0000 UTC m=+12.853981853"
Nov 03 21:14:16.661768 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:16.661703    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr" podStartSLOduration=2.683474187 podStartE2EDuration="12.661690526s" totalImagesPullingTime="9.978216339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:16.661303761 +0000 UTC m=+12.959930543" watchObservedRunningTime="2026-11-03 21:14:16.661690526 +0000 UTC m=+12.960317302"
Nov 03 21:14:17.186929 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:17.186645    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:17.187092 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:17.186702    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:17.187092 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:17.186754    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:17.187092 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:17.187006    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xvgt" podUID="4b84ba29-a116-48ef-8d9f-d81ceb4b3359"
Nov 03 21:14:17.187092 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:17.187065    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5vrf" podUID="96fee653-58b4-46bf-9ed5-0d3c5a93e8a8"
Nov 03 21:14:17.187231 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:17.187114    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8w5st" podUID="784231bb-189f-472c-83ed-6ced092f10a1"
Nov 03 21:14:17.323842 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:17.323805    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6zndc" event={"ID":"23c56cf7-2a99-4df8-841a-d84b6a9a3822","Type":"ContainerStarted","Data":"8e6d9eabf5f0b64b3393d180081a7c0fd365a7dc4484dafd99a05abaffc5cae9"}
Nov 03 21:14:18.330404 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:18.330360    2574 generic.go:356] "Generic (PLEG): container finished" podID="abf1a25c-1da1-449d-90c0-35d6fc406647" containerID="11af2e37c817ac42d2d9fc208d673eba62ea9f605ef2f2307a4126b3acd77601" exitCode=0
Nov 03 21:14:18.331016 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:18.330438    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2pgd" event={"ID":"abf1a25c-1da1-449d-90c0-35d6fc406647","Type":"ContainerDied","Data":"11af2e37c817ac42d2d9fc208d673eba62ea9f605ef2f2307a4126b3acd77601"}
Nov 03 21:14:18.448712 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:18.448657    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6zndc" podStartSLOduration=3.333040109 podStartE2EDuration="14.448643707s" totalImagesPullingTime="11.115603598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:17.441065973 +0000 UTC m=+13.739692753" watchObservedRunningTime="2026-11-03 21:14:18.448643707 +0000 UTC m=+14.747270487"
Nov 03 21:14:19.186740 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:19.186673    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:19.186904 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:19.186743    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:19.186904 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:19.186764    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:19.186904 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:19.186860    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xvgt" podUID="4b84ba29-a116-48ef-8d9f-d81ceb4b3359"
Nov 03 21:14:19.187027 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:19.186947    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8w5st" podUID="784231bb-189f-472c-83ed-6ced092f10a1"
Nov 03 21:14:19.187085 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:19.187041    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5vrf" podUID="96fee653-58b4-46bf-9ed5-0d3c5a93e8a8"
Nov 03 21:14:20.338074 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:20.338046    2574 generic.go:356] "Generic (PLEG): container finished" podID="abf1a25c-1da1-449d-90c0-35d6fc406647" containerID="3297410a43be62f39b61871bd6d6a8dd5aa0fff569ea925cab41f01be6d346f2" exitCode=0
Nov 03 21:14:20.338624 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:20.338085    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2pgd" event={"ID":"abf1a25c-1da1-449d-90c0-35d6fc406647","Type":"ContainerDied","Data":"3297410a43be62f39b61871bd6d6a8dd5aa0fff569ea925cab41f01be6d346f2"}
Nov 03 21:14:21.147891 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:21.147868    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-original-pull-secret\") pod \"global-pull-secret-syncer-v5vrf\" (UID: \"96fee653-58b4-46bf-9ed5-0d3c5a93e8a8\") " pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:21.147974 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:21.147908    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs\") pod \"network-metrics-daemon-6xvgt\" (UID: \"4b84ba29-a116-48ef-8d9f-d81ceb4b3359\") " pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:21.148028 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:21.148008    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:21.148028 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:21.148008    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:21.148090 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:21.148066    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs podName:4b84ba29-a116-48ef-8d9f-d81ceb4b3359 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:37.148049281 +0000 UTC m=+33.446676054 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs") pod "network-metrics-daemon-6xvgt" (UID: "4b84ba29-a116-48ef-8d9f-d81ceb4b3359") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:21.148090 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:21.148079    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-original-pull-secret podName:96fee653-58b4-46bf-9ed5-0d3c5a93e8a8 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:37.148073083 +0000 UTC m=+33.446699841 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-original-pull-secret") pod "global-pull-secret-syncer-v5vrf" (UID: "96fee653-58b4-46bf-9ed5-0d3c5a93e8a8") : object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:21.187608 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:21.187402    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:21.187718 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:21.187439    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:21.187718 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:21.187447    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:21.187791 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:21.187737    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xvgt" podUID="4b84ba29-a116-48ef-8d9f-d81ceb4b3359"
Nov 03 21:14:21.187848 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:21.187830    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8w5st" podUID="784231bb-189f-472c-83ed-6ced092f10a1"
Nov 03 21:14:21.187926 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:21.187909    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5vrf" podUID="96fee653-58b4-46bf-9ed5-0d3c5a93e8a8"
Nov 03 21:14:21.248825 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:21.248798    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb9kd\" (UniqueName: \"kubernetes.io/projected/784231bb-189f-472c-83ed-6ced092f10a1-kube-api-access-nb9kd\") pod \"network-check-target-8w5st\" (UID: \"784231bb-189f-472c-83ed-6ced092f10a1\") " pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:21.248969 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:21.248953    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 21:14:21.249003 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:21.248993    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 21:14:21.249043 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:21.249003    2574 projected.go:196] Error preparing data for projected volume kube-api-access-nb9kd for pod openshift-network-diagnostics/network-check-target-8w5st: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:21.249075 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:21.249054    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/784231bb-189f-472c-83ed-6ced092f10a1-kube-api-access-nb9kd podName:784231bb-189f-472c-83ed-6ced092f10a1 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:37.249038896 +0000 UTC m=+33.547665668 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nb9kd" (UniqueName: "kubernetes.io/projected/784231bb-189f-472c-83ed-6ced092f10a1-kube-api-access-nb9kd") pod "network-check-target-8w5st" (UID: "784231bb-189f-472c-83ed-6ced092f10a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:21.342658 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:21.342622    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6" event={"ID":"a295913f-3811-47d6-9e85-a76182e45649","Type":"ContainerStarted","Data":"79d3b0235001a3a1ab99925d06b325fcab0760565afa681654a8a0ed6b18e50c"}
Nov 03 21:14:21.343066 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:21.342735    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6" event={"ID":"a295913f-3811-47d6-9e85-a76182e45649","Type":"ContainerStarted","Data":"3e8e9ba661ea2e556103b6ffc960d66dd63ab4f9849a10dc5eedacaca13e6065"}
Nov 03 21:14:22.348553 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:22.348331    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2pgd" event={"ID":"abf1a25c-1da1-449d-90c0-35d6fc406647","Type":"ContainerStarted","Data":"4fc59f451eac6beab6e223e05687434c39bbff6d9fb541a52c5ba5bf86ec28ac"}
Nov 03 21:14:22.351208 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:22.351185    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:14:22.351602 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:22.351580    2574 generic.go:356] "Generic (PLEG): container finished" podID="a295913f-3811-47d6-9e85-a76182e45649" containerID="79d3b0235001a3a1ab99925d06b325fcab0760565afa681654a8a0ed6b18e50c" exitCode=1
Nov 03 21:14:22.351666 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:22.351618    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6" event={"ID":"a295913f-3811-47d6-9e85-a76182e45649","Type":"ContainerDied","Data":"79d3b0235001a3a1ab99925d06b325fcab0760565afa681654a8a0ed6b18e50c"}
Nov 03 21:14:22.351701 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:22.351673    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6" event={"ID":"a295913f-3811-47d6-9e85-a76182e45649","Type":"ContainerStarted","Data":"2b1957284eabfe41586d18308afda48e1799b25b0f61df995c5b42fd7fa8ade4"}
Nov 03 21:14:22.351732 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:22.351703    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6" event={"ID":"a295913f-3811-47d6-9e85-a76182e45649","Type":"ContainerStarted","Data":"feff799b9a2ba7d0bc87f09b1121c645e3db0a3cac6bd82fcbf380d47f1c0d39"}
Nov 03 21:14:22.351760 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:22.351732    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6" event={"ID":"a295913f-3811-47d6-9e85-a76182e45649","Type":"ContainerStarted","Data":"9494099c4e49d615e52c95817711b291af15a34134d77836c06ae5bab2272aeb"}
Nov 03 21:14:22.351793 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:22.351759    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6" event={"ID":"a295913f-3811-47d6-9e85-a76182e45649","Type":"ContainerStarted","Data":"5e5cc2d390c7be2ea98242c2a9ba5a45b726e59edcdbab52e9b07f3fc576b738"}
Nov 03 21:14:23.187430 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:23.187398    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:23.187430 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:23.187431    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:23.187627 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:23.187398    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:23.187627 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:23.187494    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8w5st" podUID="784231bb-189f-472c-83ed-6ced092f10a1"
Nov 03 21:14:23.187627 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:23.187594    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5vrf" podUID="96fee653-58b4-46bf-9ed5-0d3c5a93e8a8"
Nov 03 21:14:23.187713 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:23.187673    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xvgt" podUID="4b84ba29-a116-48ef-8d9f-d81ceb4b3359"
Nov 03 21:14:23.356290 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:23.356259    2574 generic.go:356] "Generic (PLEG): container finished" podID="abf1a25c-1da1-449d-90c0-35d6fc406647" containerID="4fc59f451eac6beab6e223e05687434c39bbff6d9fb541a52c5ba5bf86ec28ac" exitCode=0
Nov 03 21:14:23.356659 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:23.356317    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2pgd" event={"ID":"abf1a25c-1da1-449d-90c0-35d6fc406647","Type":"ContainerDied","Data":"4fc59f451eac6beab6e223e05687434c39bbff6d9fb541a52c5ba5bf86ec28ac"}
Nov 03 21:14:24.361223 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:24.361191    2574 generic.go:356] "Generic (PLEG): container finished" podID="abf1a25c-1da1-449d-90c0-35d6fc406647" containerID="41a075e0982a2efef1cd88f6ae89ac3bc5581d050ba45031e232c03648799294" exitCode=0
Nov 03 21:14:24.361767 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:24.361286    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2pgd" event={"ID":"abf1a25c-1da1-449d-90c0-35d6fc406647","Type":"ContainerDied","Data":"41a075e0982a2efef1cd88f6ae89ac3bc5581d050ba45031e232c03648799294"}
Nov 03 21:14:24.364503 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:24.364484    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:14:24.364836 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:24.364816    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6" event={"ID":"a295913f-3811-47d6-9e85-a76182e45649","Type":"ContainerStarted","Data":"1957d26fe3d82aed2203c8d132dd91c1e450630e6792deec8474baf5902c6ff0"}
Nov 03 21:14:24.547486 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:24.547448    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cccr5"
Nov 03 21:14:24.547687 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:24.547598    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 21:14:24.548021 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:24.548001    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cccr5"
Nov 03 21:14:25.187480 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:25.187448    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:25.187480 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:25.187479    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:25.187684 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:25.187548    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xvgt" podUID="4b84ba29-a116-48ef-8d9f-d81ceb4b3359"
Nov 03 21:14:25.187684 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:25.187608    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:25.187754 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:25.187697    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5vrf" podUID="96fee653-58b4-46bf-9ed5-0d3c5a93e8a8"
Nov 03 21:14:25.187783 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:25.187766    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8w5st" podUID="784231bb-189f-472c-83ed-6ced092f10a1"
Nov 03 21:14:25.370384 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:25.370347    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2pgd" event={"ID":"abf1a25c-1da1-449d-90c0-35d6fc406647","Type":"ContainerStarted","Data":"09192c8b64ba4fa4101e74ab160421227afa2e6cca6e107d217223c53b4be96a"}
Nov 03 21:14:25.488053 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:25.488004    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-f2pgd" podStartSLOduration=9.088431544 podStartE2EDuration="21.487991633s" totalImagesPullingTime="12.399560089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:25.487735761 +0000 UTC m=+21.786362540" watchObservedRunningTime="2026-11-03 21:14:25.487991633 +0000 UTC m=+21.786618395"
Nov 03 21:14:26.376552 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:26.376391    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:14:26.377112 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:26.376859    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6" event={"ID":"a295913f-3811-47d6-9e85-a76182e45649","Type":"ContainerStarted","Data":"12167c7edaac921c724c071695060dc83215ddb061fb2c28d61b8dcce3b60835"}
Nov 03 21:14:26.377491 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:26.377476    2574 scope.go:122] "RemoveContainer" containerID="79d3b0235001a3a1ab99925d06b325fcab0760565afa681654a8a0ed6b18e50c"
Nov 03 21:14:27.187373 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:27.187337    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:27.187720 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:27.187345    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:27.187720 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:27.187449    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xvgt" podUID="4b84ba29-a116-48ef-8d9f-d81ceb4b3359"
Nov 03 21:14:27.187720 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:27.187485    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:27.187720 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:27.187610    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8w5st" podUID="784231bb-189f-472c-83ed-6ced092f10a1"
Nov 03 21:14:27.187720 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:27.187693    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5vrf" podUID="96fee653-58b4-46bf-9ed5-0d3c5a93e8a8"
Nov 03 21:14:27.382510 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:27.382483    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:14:27.382920 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:27.382787    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6" event={"ID":"a295913f-3811-47d6-9e85-a76182e45649","Type":"ContainerStarted","Data":"8b05db7d766c32673554bf4054019d3a4fa5a2dcf99eb36cbb3ee3757b29d0f0"}
Nov 03 21:14:27.382963 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:27.382934    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 21:14:27.383287 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:27.383270    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:27.383385 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:27.383341    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:27.398734 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:27.398709    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:27.398889 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:27.398856    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:27.639814 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:27.639747    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6" podStartSLOduration=8.218112158 podStartE2EDuration="23.639728575s" totalImagesPullingTime="15.421616417s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:27.606454792 +0000 UTC m=+23.905081572" watchObservedRunningTime="2026-11-03 21:14:27.639728575 +0000 UTC m=+23.938355380"
Nov 03 21:14:28.385812 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:28.385782    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 21:14:28.443560 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:28.443530    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v5vrf"]
Nov 03 21:14:28.443717 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:28.443633    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:28.443717 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:28.443707    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5vrf" podUID="96fee653-58b4-46bf-9ed5-0d3c5a93e8a8"
Nov 03 21:14:28.446927 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:28.446893    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6xvgt"]
Nov 03 21:14:28.447076 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:28.447027    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:28.447168 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:28.447141    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xvgt" podUID="4b84ba29-a116-48ef-8d9f-d81ceb4b3359"
Nov 03 21:14:28.447629 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:28.447611    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8w5st"]
Nov 03 21:14:28.447697 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:28.447687    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:28.447764 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:28.447749    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8w5st" podUID="784231bb-189f-472c-83ed-6ced092f10a1"
Nov 03 21:14:29.388080 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:29.388047    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 21:14:30.187105 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:30.187021    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:30.187317 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:30.187024    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:30.187317 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:30.187130    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8w5st" podUID="784231bb-189f-472c-83ed-6ced092f10a1"
Nov 03 21:14:30.187317 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:30.187273    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xvgt" podUID="4b84ba29-a116-48ef-8d9f-d81ceb4b3359"
Nov 03 21:14:30.187317 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:30.187311    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:30.187498 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:30.187388    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5vrf" podUID="96fee653-58b4-46bf-9ed5-0d3c5a93e8a8"
Nov 03 21:14:31.203297 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:31.203267    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:31.203843 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:31.203459    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 21:14:31.215579 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:31.215537    2574 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6" podUID="a295913f-3811-47d6-9e85-a76182e45649" containerName="ovnkube-controller" probeResult="failure" output=""
Nov 03 21:14:31.224477 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:31.224449    2574 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6" podUID="a295913f-3811-47d6-9e85-a76182e45649" containerName="ovnkube-controller" probeResult="failure" output=""
Nov 03 21:14:32.187153 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:32.187071    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:32.187153 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:32.187101    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:32.187153 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:32.187121    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:32.187449 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:32.187203    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5vrf" podUID="96fee653-58b4-46bf-9ed5-0d3c5a93e8a8"
Nov 03 21:14:32.187449 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:32.187321    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xvgt" podUID="4b84ba29-a116-48ef-8d9f-d81ceb4b3359"
Nov 03 21:14:32.187611 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:32.187444    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8w5st" podUID="784231bb-189f-472c-83ed-6ced092f10a1"
Nov 03 21:14:34.013978 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.013806    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeReady"
Nov 03 21:14:34.014404 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.014070    2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 21:14:34.131349 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.131255    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-sds64"]
Nov 03 21:14:34.133207 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.133191    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-27lr2"]
Nov 03 21:14:34.133351 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.133333    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-sds64"
Nov 03 21:14:34.134605 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.134585    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bhpkz"]
Nov 03 21:14:34.134716 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.134640    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-27lr2"
Nov 03 21:14:34.135959 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.135941    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bhpkz"
Nov 03 21:14:34.171083 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.171041    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"]
Nov 03 21:14:34.172736 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.172711    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-bc6rw"]
Nov 03 21:14:34.172890 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.172834    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:34.174215 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.174197    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:34.182387 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.182367    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5ffbbbfcf7-99qbj"]
Nov 03 21:14:34.184180 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.184162    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nv9qq"]
Nov 03 21:14:34.184312 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.184295    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:34.185777 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.185758    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"]
Nov 03 21:14:34.185898 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.185880    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nv9qq"
Nov 03 21:14:34.187896 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.187490    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:34.187896 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.187520    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:34.187896 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.187629    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:34.187896 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.187814    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:34.192340 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.192322    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8"]
Nov 03 21:14:34.193800 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.193773    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"]
Nov 03 21:14:34.193892 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.193875    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8"
Nov 03 21:14:34.195462 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.195441    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7"]
Nov 03 21:14:34.195571 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.195516    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"
Nov 03 21:14:34.197694 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.197679    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-j2ksl"]
Nov 03 21:14:34.197870 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.197850    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7"
Nov 03 21:14:34.199418 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.199401    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-j2ksl"
Nov 03 21:14:34.222051 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.222030    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6df897dfcf-2b92c"]
Nov 03 21:14:34.224065 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.224045    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75cb56744b-c6rsx"]
Nov 03 21:14:34.224210 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.224195    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.225619 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.225603    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-sb5xl"]
Nov 03 21:14:34.225734 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.225720    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.227275 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.227259    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8p8hk"]
Nov 03 21:14:34.227367 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.227315    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sb5xl"
Nov 03 21:14:34.231819 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.231794    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:34.232439 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.232420    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 21:14:34.232823 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.232805    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 21:14:34.234496 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.234480    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-m5s9b"]
Nov 03 21:14:34.236166 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.236143    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 21:14:34.240387 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.240335    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bhpkz"]
Nov 03 21:14:34.240387 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.240381    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-sds64"]
Nov 03 21:14:34.240747 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.240405    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-27lr2"]
Nov 03 21:14:34.240747 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.240492    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m5s9b"
Nov 03 21:14:34.241053 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241026    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjp6p\" (UniqueName: \"kubernetes.io/projected/067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa-kube-api-access-fjp6p\") pod \"service-ca-operator-544d749c8c-k65s8\" (UID: \"067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8"
Nov 03 21:14:34.241139 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241065    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-default-certificate\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:34.241139 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241092    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-stats-auth\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:34.241305 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241194    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvsxw\" (UniqueName: \"kubernetes.io/projected/35650ff5-011e-4572-886f-8bf47f598219-kube-api-access-gvsxw\") pod \"volume-data-source-validator-58c9f5f46d-nv9qq\" (UID: \"35650ff5-011e-4572-886f-8bf47f598219\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nv9qq"
Nov 03 21:14:34.241305 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241221    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-bc6rw"]
Nov 03 21:14:34.241305 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241250    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fphb6\" (UniqueName: \"kubernetes.io/projected/5cacf547-c691-4b14-936c-e6b08691bf93-kube-api-access-fphb6\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:34.241305 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241297    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5cacf547-c691-4b14-936c-e6b08691bf93-snapshots\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:34.241544 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241323    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:34.241544 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241393    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jf555\" (UID: \"ae204c86-8d93-4adf-988c-cd836e011270\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:34.241544 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241416    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzr7h\" (UniqueName: \"kubernetes.io/projected/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-kube-api-access-fzr7h\") pod \"cluster-samples-operator-6ddb6c8b94-sqlqz\" (UID: \"cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"
Nov 03 21:14:34.241544 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241437    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cacf547-c691-4b14-936c-e6b08691bf93-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:34.241544 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241518    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6547c71f-71a2-40c9-912f-9283b6bdb560-trusted-ca\") pod \"console-operator-7cd56b8dbc-rhpjv\" (UID: \"6547c71f-71a2-40c9-912f-9283b6bdb560\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:34.241544 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241544    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cacf547-c691-4b14-936c-e6b08691bf93-service-ca-bundle\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:34.241544 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241565    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5cacf547-c691-4b14-936c-e6b08691bf93-serving-cert\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:34.241932 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241589    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa-config\") pod \"service-ca-operator-544d749c8c-k65s8\" (UID: \"067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8"
Nov 03 21:14:34.241932 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241609    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cacf547-c691-4b14-936c-e6b08691bf93-tmp\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:34.241932 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241633    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sqlqz\" (UID: \"cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"
Nov 03 21:14:34.241932 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241655    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zn62\" (UniqueName: \"kubernetes.io/projected/974ed564-6386-4e1e-9326-f14ccc36658c-kube-api-access-4zn62\") pod \"kas-connection-checker-685895c7c9-27lr2\" (UID: \"974ed564-6386-4e1e-9326-f14ccc36658c\") " pod="kube-system/kas-connection-checker-685895c7c9-27lr2"
Nov 03 21:14:34.241932 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241686    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6547c71f-71a2-40c9-912f-9283b6bdb560-config\") pod \"console-operator-7cd56b8dbc-rhpjv\" (UID: \"6547c71f-71a2-40c9-912f-9283b6bdb560\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:34.241932 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241706    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbv62\" (UniqueName: \"kubernetes.io/projected/6547c71f-71a2-40c9-912f-9283b6bdb560-kube-api-access-qbv62\") pod \"console-operator-7cd56b8dbc-rhpjv\" (UID: \"6547c71f-71a2-40c9-912f-9283b6bdb560\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:34.241932 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241734    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgjr7\" (UniqueName: \"kubernetes.io/projected/52709fc1-b443-4a46-a95e-20cacc4ac4a0-kube-api-access-qgjr7\") pod \"kas-connection-checker-685895c7c9-sds64\" (UID: \"52709fc1-b443-4a46-a95e-20cacc4ac4a0\") " pod="kube-system/kas-connection-checker-685895c7c9-sds64"
Nov 03 21:14:34.241932 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241755    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcxtm\" (UniqueName: \"kubernetes.io/projected/ae204c86-8d93-4adf-988c-cd836e011270-kube-api-access-hcxtm\") pod \"cluster-monitoring-operator-884576c57-jf555\" (UID: \"ae204c86-8d93-4adf-988c-cd836e011270\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:34.241932 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241777    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42cwb\" (UniqueName: \"kubernetes.io/projected/bbd3c724-e7e5-4382-b681-adf7fd8f302b-kube-api-access-42cwb\") pod \"kas-connection-checker-685895c7c9-bhpkz\" (UID: \"bbd3c724-e7e5-4382-b681-adf7fd8f302b\") " pod="kube-system/kas-connection-checker-685895c7c9-bhpkz"
Nov 03 21:14:34.241932 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241809    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:34.241932 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241831    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6547c71f-71a2-40c9-912f-9283b6bdb560-serving-cert\") pod \"console-operator-7cd56b8dbc-rhpjv\" (UID: \"6547c71f-71a2-40c9-912f-9283b6bdb560\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:34.241932 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241858    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ae204c86-8d93-4adf-988c-cd836e011270-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jf555\" (UID: \"ae204c86-8d93-4adf-988c-cd836e011270\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:34.241932 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241881    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-sqlqz\" (UID: \"cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"
Nov 03 21:14:34.241932 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241905    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs262\" (UniqueName: \"kubernetes.io/projected/7ccca940-de12-451b-b920-d6bef516942f-kube-api-access-fs262\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:34.241932 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.241932    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa-serving-cert\") pod \"service-ca-operator-544d749c8c-k65s8\" (UID: \"067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8"
Nov 03 21:14:34.245610 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.245588    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5ffbbbfcf7-99qbj"]
Nov 03 21:14:34.252908 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.252773    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-j2ksl"]
Nov 03 21:14:34.257038 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.257004    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"]
Nov 03 21:14:34.257882 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.257856    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6df897dfcf-2b92c"]
Nov 03 21:14:34.293256 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.293211    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 21:14:34.293430 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.293270    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 21:14:34.293430 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.293309    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 21:14:34.293430 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.293313    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 21:14:34.293430 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.293381    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 21:14:34.293430 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.293211    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 21:14:34.293430 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.293383    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 21:14:34.293430 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.293389    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 21:14:34.293705 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.293449    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 21:14:34.293705 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.293473    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 21:14:34.294138 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.294117    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 21:14:34.294275 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.294170    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 21:14:34.294275 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.294270    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 21:14:34.294403 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.294291    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 21:14:34.294403 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.294342    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 21:14:34.294403 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.294365    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 21:14:34.294554 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.294440    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 21:14:34.294554 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.294443    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 21:14:34.294554 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.294502    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 21:14:34.294554 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.294528    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 21:14:34.294846 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.294589    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 21:14:34.294846 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.294613    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 21:14:34.294846 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.294682    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 21:14:34.294846 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.294703    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 21:14:34.294846 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.294765    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 21:14:34.298982 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.295785    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m5s9b"]
Nov 03 21:14:34.298982 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.297378    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 21:14:34.342500 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.342465    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fphb6\" (UniqueName: \"kubernetes.io/projected/5cacf547-c691-4b14-936c-e6b08691bf93-kube-api-access-fphb6\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:34.342500 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.342499    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:34.342698 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.342527    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j2ksl\" (UID: \"bb8a800c-36be-4da0-b1e6-5ef65f994fc1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j2ksl"
Nov 03 21:14:34.342698 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.342661    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle podName:7ccca940-de12-451b-b920-d6bef516942f nodeName:}" failed. No retries permitted until 2026-11-03 21:14:34.842635558 +0000 UTC m=+31.141262338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle") pod "router-default-5ffbbbfcf7-99qbj" (UID: "7ccca940-de12-451b-b920-d6bef516942f") : configmap references non-existent config key: service-ca.crt
Nov 03 21:14:34.342826 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.342808    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ef7a859e-70f5-497a-9ad4-6614885fb71e-ca-trust-extracted\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.342879 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.342842    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5cacf547-c691-4b14-936c-e6b08691bf93-snapshots\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:34.342879 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.342860    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-installation-pull-secrets\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.342978 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.342879    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jf555\" (UID: \"ae204c86-8d93-4adf-988c-cd836e011270\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:34.342978 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.342902    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzr7h\" (UniqueName: \"kubernetes.io/projected/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-kube-api-access-fzr7h\") pod \"cluster-samples-operator-6ddb6c8b94-sqlqz\" (UID: \"cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"
Nov 03 21:14:34.344200 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343101    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cacf547-c691-4b14-936c-e6b08691bf93-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:34.344200 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.343133    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 21:14:34.344200 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343143    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m22kb\" (UniqueName: \"kubernetes.io/projected/eea6e5c2-57f8-42c6-9ce9-55a3c8f92790-kube-api-access-m22kb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7\" (UID: \"eea6e5c2-57f8-42c6-9ce9-55a3c8f92790\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7"
Nov 03 21:14:34.344200 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343173    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-image-registry-private-configuration\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.344200 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.343191    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls podName:ae204c86-8d93-4adf-988c-cd836e011270 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:34.843172296 +0000 UTC m=+31.141799069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jf555" (UID: "ae204c86-8d93-4adf-988c-cd836e011270") : secret "cluster-monitoring-operator-tls" not found
Nov 03 21:14:34.344200 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343224    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhdwv\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-kube-api-access-bhdwv\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.344200 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343307    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6547c71f-71a2-40c9-912f-9283b6bdb560-trusted-ca\") pod \"console-operator-7cd56b8dbc-rhpjv\" (UID: \"6547c71f-71a2-40c9-912f-9283b6bdb560\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:34.344200 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343339    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-certificates\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.344200 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343368    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5cacf547-c691-4b14-936c-e6b08691bf93-serving-cert\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:34.344200 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343392    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6547c71f-71a2-40c9-912f-9283b6bdb560-config\") pod \"console-operator-7cd56b8dbc-rhpjv\" (UID: \"6547c71f-71a2-40c9-912f-9283b6bdb560\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:34.344200 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343413    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eea6e5c2-57f8-42c6-9ce9-55a3c8f92790-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7\" (UID: \"eea6e5c2-57f8-42c6-9ce9-55a3c8f92790\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7"
Nov 03 21:14:34.344200 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343438    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sqlqz\" (UID: \"cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"
Nov 03 21:14:34.344200 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343461    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ef7a859e-70f5-497a-9ad4-6614885fb71e-image-registry-private-configuration\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.344200 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343487    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbv62\" (UniqueName: \"kubernetes.io/projected/6547c71f-71a2-40c9-912f-9283b6bdb560-kube-api-access-qbv62\") pod \"console-operator-7cd56b8dbc-rhpjv\" (UID: \"6547c71f-71a2-40c9-912f-9283b6bdb560\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:34.344200 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343508    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-bound-sa-token\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.344978 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343536    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgjr7\" (UniqueName: \"kubernetes.io/projected/52709fc1-b443-4a46-a95e-20cacc4ac4a0-kube-api-access-qgjr7\") pod \"kas-connection-checker-685895c7c9-sds64\" (UID: \"52709fc1-b443-4a46-a95e-20cacc4ac4a0\") " pod="kube-system/kas-connection-checker-685895c7c9-sds64"
Nov 03 21:14:34.344978 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343554    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42cwb\" (UniqueName: \"kubernetes.io/projected/bbd3c724-e7e5-4382-b681-adf7fd8f302b-kube-api-access-42cwb\") pod \"kas-connection-checker-685895c7c9-bhpkz\" (UID: \"bbd3c724-e7e5-4382-b681-adf7fd8f302b\") " pod="kube-system/kas-connection-checker-685895c7c9-bhpkz"
Nov 03 21:14:34.344978 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343570    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-config-volume\") pod \"dns-default-8p8hk\" (UID: \"aadb342f-3310-4c81-967d-0f0a8ef5a4e6\") " pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:34.344978 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343592    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c27ff312-c8d3-4691-b535-ea8adf44d9c3-cert\") pod \"ingress-canary-m5s9b\" (UID: \"c27ff312-c8d3-4691-b535-ea8adf44d9c3\") " pod="openshift-ingress-canary/ingress-canary-m5s9b"
Nov 03 21:14:34.344978 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343610    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:34.344978 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343635    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6547c71f-71a2-40c9-912f-9283b6bdb560-serving-cert\") pod \"console-operator-7cd56b8dbc-rhpjv\" (UID: \"6547c71f-71a2-40c9-912f-9283b6bdb560\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:34.344978 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343665    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ae204c86-8d93-4adf-988c-cd836e011270-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jf555\" (UID: \"ae204c86-8d93-4adf-988c-cd836e011270\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:34.344978 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343689    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-sqlqz\" (UID: \"cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"
Nov 03 21:14:34.344978 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343717    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs262\" (UniqueName: \"kubernetes.io/projected/7ccca940-de12-451b-b920-d6bef516942f-kube-api-access-fs262\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:34.344978 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343740    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-trusted-ca\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.344978 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343748    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5cacf547-c691-4b14-936c-e6b08691bf93-snapshots\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:34.344978 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343764    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa-serving-cert\") pod \"service-ca-operator-544d749c8c-k65s8\" (UID: \"067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8"
Nov 03 21:14:34.344978 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343793    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjp6p\" (UniqueName: \"kubernetes.io/projected/067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa-kube-api-access-fjp6p\") pod \"service-ca-operator-544d749c8c-k65s8\" (UID: \"067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8"
Nov 03 21:14:34.344978 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343818    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-default-certificate\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:34.344978 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343847    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-stats-auth\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:34.344978 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343872    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv8v4\" (UniqueName: \"kubernetes.io/projected/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-kube-api-access-pv8v4\") pod \"dns-default-8p8hk\" (UID: \"aadb342f-3310-4c81-967d-0f0a8ef5a4e6\") " pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:34.345781 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343902    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvsxw\" (UniqueName: \"kubernetes.io/projected/35650ff5-011e-4572-886f-8bf47f598219-kube-api-access-gvsxw\") pod \"volume-data-source-validator-58c9f5f46d-nv9qq\" (UID: \"35650ff5-011e-4572-886f-8bf47f598219\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nv9qq"
Nov 03 21:14:34.345781 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343922    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-ca-trust-extracted\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.345781 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343942    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpzsk\" (UniqueName: \"kubernetes.io/projected/c27ff312-c8d3-4691-b535-ea8adf44d9c3-kube-api-access-zpzsk\") pod \"ingress-canary-m5s9b\" (UID: \"c27ff312-c8d3-4691-b535-ea8adf44d9c3\") " pod="openshift-ingress-canary/ingress-canary-m5s9b"
Nov 03 21:14:34.345781 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343962    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j2ksl\" (UID: \"bb8a800c-36be-4da0-b1e6-5ef65f994fc1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j2ksl"
Nov 03 21:14:34.345781 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343999    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.345781 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.344026    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jrqs\" (UniqueName: \"kubernetes.io/projected/06a672d7-0667-4bab-98ee-91dd71dfc489-kube-api-access-9jrqs\") pod \"network-check-source-6f78d59f99-sb5xl\" (UID: \"06a672d7-0667-4bab-98ee-91dd71dfc489\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sb5xl"
Nov 03 21:14:34.345781 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.344107    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-certificates\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.345781 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.344154    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-metrics-tls\") pod \"dns-default-8p8hk\" (UID: \"aadb342f-3310-4c81-967d-0f0a8ef5a4e6\") " pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:34.345781 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.343637    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 21:14:34.345781 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.344298    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6547c71f-71a2-40c9-912f-9283b6bdb560-config\") pod \"console-operator-7cd56b8dbc-rhpjv\" (UID: \"6547c71f-71a2-40c9-912f-9283b6bdb560\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:34.345781 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.344347    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-tmp-dir\") pod \"dns-default-8p8hk\" (UID: \"aadb342f-3310-4c81-967d-0f0a8ef5a4e6\") " pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:34.345781 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.344437    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cacf547-c691-4b14-936c-e6b08691bf93-service-ca-bundle\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:34.345781 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.344473    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ef7a859e-70f5-497a-9ad4-6614885fb71e-installation-pull-secrets\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.345781 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.344911    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa-config\") pod \"service-ca-operator-544d749c8c-k65s8\" (UID: \"067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8"
Nov 03 21:14:34.345781 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.344942    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cacf547-c691-4b14-936c-e6b08691bf93-tmp\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:34.345781 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.344969    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cacf547-c691-4b14-936c-e6b08691bf93-service-ca-bundle\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:34.346379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.344970    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.346379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.345041    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zn62\" (UniqueName: \"kubernetes.io/projected/974ed564-6386-4e1e-9326-f14ccc36658c-kube-api-access-4zn62\") pod \"kas-connection-checker-685895c7c9-27lr2\" (UID: \"974ed564-6386-4e1e-9326-f14ccc36658c\") " pod="kube-system/kas-connection-checker-685895c7c9-27lr2"
Nov 03 21:14:34.346379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.345055    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-sqlqz\" (UID: \"cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"
Nov 03 21:14:34.346379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.345075    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp68p\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-kube-api-access-lp68p\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.346379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.345103    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef7a859e-70f5-497a-9ad4-6614885fb71e-trusted-ca\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.346379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.345045    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ae204c86-8d93-4adf-988c-cd836e011270-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jf555\" (UID: \"ae204c86-8d93-4adf-988c-cd836e011270\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:34.346379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.345282    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cacf547-c691-4b14-936c-e6b08691bf93-tmp\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:34.346379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.345299    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-sb5xl"]
Nov 03 21:14:34.346379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.345336    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcxtm\" (UniqueName: \"kubernetes.io/projected/ae204c86-8d93-4adf-988c-cd836e011270-kube-api-access-hcxtm\") pod \"cluster-monitoring-operator-884576c57-jf555\" (UID: \"ae204c86-8d93-4adf-988c-cd836e011270\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:34.346379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.345379    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eea6e5c2-57f8-42c6-9ce9-55a3c8f92790-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7\" (UID: \"eea6e5c2-57f8-42c6-9ce9-55a3c8f92790\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7"
Nov 03 21:14:34.346379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.345400    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-bound-sa-token\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.346379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.345640    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa-config\") pod \"service-ca-operator-544d749c8c-k65s8\" (UID: \"067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8"
Nov 03 21:14:34.346697 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.346616    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8"]
Nov 03 21:14:34.347624 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.347607    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"]
Nov 03 21:14:34.348159 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.348145    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"]
Nov 03 21:14:34.394773 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.394670    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 21:14:34.394980 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.394825    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 21:14:34.395414 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.395395    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rbksq\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.395797    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.395900    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.396005    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.396065    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.396322    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.396351    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.396396    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.396657    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.396667    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.396686    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-b97xq\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.397140    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-25rf5\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.397169    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.397211    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jl8l5\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.397270    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.397525    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l78m6\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.397604    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.397625    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jnct2\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.397753    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.397793    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.397803    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.397809    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.397834    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.397604    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.397978    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.398049    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs podName:7ccca940-de12-451b-b920-d6bef516942f nodeName:}" failed. No retries permitted until 2026-11-03 21:14:34.898028885 +0000 UTC m=+31.196655660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs") pod "router-default-5ffbbbfcf7-99qbj" (UID: "7ccca940-de12-451b-b920-d6bef516942f") : secret "router-metrics-certs-default" not found
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.398258    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6d8l7\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.398319    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-l8m5w\""
Nov 03 21:14:34.398661 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.398348    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 21:14:34.400866 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.400844    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nv9qq"]
Nov 03 21:14:34.401858 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.401833    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75cb56744b-c6rsx"]
Nov 03 21:14:34.403689 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.403291    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6547c71f-71a2-40c9-912f-9283b6bdb560-serving-cert\") pod \"console-operator-7cd56b8dbc-rhpjv\" (UID: \"6547c71f-71a2-40c9-912f-9283b6bdb560\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:34.403777 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.403756    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 21:14:34.403830 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.403817    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-tls podName:cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c nodeName:}" failed. No retries permitted until 2026-11-03 21:14:34.903798327 +0000 UTC m=+31.202425102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-sqlqz" (UID: "cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c") : secret "samples-operator-tls" not found
Nov 03 21:14:34.403961 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.403935    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7"]
Nov 03 21:14:34.405437 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.404320    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fphb6\" (UniqueName: \"kubernetes.io/projected/5cacf547-c691-4b14-936c-e6b08691bf93-kube-api-access-fphb6\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:34.405437 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.404888    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cacf547-c691-4b14-936c-e6b08691bf93-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:34.405437 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.404956    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8p8hk"]
Nov 03 21:14:34.405437 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.405060    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6547c71f-71a2-40c9-912f-9283b6bdb560-trusted-ca\") pod \"console-operator-7cd56b8dbc-rhpjv\" (UID: \"6547c71f-71a2-40c9-912f-9283b6bdb560\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:34.405437 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.405169    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzr7h\" (UniqueName: \"kubernetes.io/projected/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-kube-api-access-fzr7h\") pod \"cluster-samples-operator-6ddb6c8b94-sqlqz\" (UID: \"cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"
Nov 03 21:14:34.405731 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.405527    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbv62\" (UniqueName: \"kubernetes.io/projected/6547c71f-71a2-40c9-912f-9283b6bdb560-kube-api-access-qbv62\") pod \"console-operator-7cd56b8dbc-rhpjv\" (UID: \"6547c71f-71a2-40c9-912f-9283b6bdb560\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:34.405731 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.405670    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvsxw\" (UniqueName: \"kubernetes.io/projected/35650ff5-011e-4572-886f-8bf47f598219-kube-api-access-gvsxw\") pod \"volume-data-source-validator-58c9f5f46d-nv9qq\" (UID: \"35650ff5-011e-4572-886f-8bf47f598219\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nv9qq"
Nov 03 21:14:34.407205 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.407176    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zn62\" (UniqueName: \"kubernetes.io/projected/974ed564-6386-4e1e-9326-f14ccc36658c-kube-api-access-4zn62\") pod \"kas-connection-checker-685895c7c9-27lr2\" (UID: \"974ed564-6386-4e1e-9326-f14ccc36658c\") " pod="kube-system/kas-connection-checker-685895c7c9-27lr2"
Nov 03 21:14:34.407417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.407392    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42cwb\" (UniqueName: \"kubernetes.io/projected/bbd3c724-e7e5-4382-b681-adf7fd8f302b-kube-api-access-42cwb\") pod \"kas-connection-checker-685895c7c9-bhpkz\" (UID: \"bbd3c724-e7e5-4382-b681-adf7fd8f302b\") " pod="kube-system/kas-connection-checker-685895c7c9-bhpkz"
Nov 03 21:14:34.407556 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.407532    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgjr7\" (UniqueName: \"kubernetes.io/projected/52709fc1-b443-4a46-a95e-20cacc4ac4a0-kube-api-access-qgjr7\") pod \"kas-connection-checker-685895c7c9-sds64\" (UID: \"52709fc1-b443-4a46-a95e-20cacc4ac4a0\") " pod="kube-system/kas-connection-checker-685895c7c9-sds64"
Nov 03 21:14:34.407764 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.407742    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcxtm\" (UniqueName: \"kubernetes.io/projected/ae204c86-8d93-4adf-988c-cd836e011270-kube-api-access-hcxtm\") pod \"cluster-monitoring-operator-884576c57-jf555\" (UID: \"ae204c86-8d93-4adf-988c-cd836e011270\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:34.407999 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.407973    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-stats-auth\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:34.408165 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.408140    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-default-certificate\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:34.408515 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.408494    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs262\" (UniqueName: \"kubernetes.io/projected/7ccca940-de12-451b-b920-d6bef516942f-kube-api-access-fs262\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:34.408725 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.408709    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5cacf547-c691-4b14-936c-e6b08691bf93-serving-cert\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:34.408965 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.408939    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa-serving-cert\") pod \"service-ca-operator-544d749c8c-k65s8\" (UID: \"067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8"
Nov 03 21:14:34.409853 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.409834    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjp6p\" (UniqueName: \"kubernetes.io/projected/067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa-kube-api-access-fjp6p\") pod \"service-ca-operator-544d749c8c-k65s8\" (UID: \"067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8"
Nov 03 21:14:34.446228 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.446183    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-sds64"
Nov 03 21:14:34.446424 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.446403    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j2ksl\" (UID: \"bb8a800c-36be-4da0-b1e6-5ef65f994fc1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j2ksl"
Nov 03 21:14:34.446464 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.446434    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ef7a859e-70f5-497a-9ad4-6614885fb71e-ca-trust-extracted\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.446464 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.446453    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-installation-pull-secrets\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.446527 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.446480    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m22kb\" (UniqueName: \"kubernetes.io/projected/eea6e5c2-57f8-42c6-9ce9-55a3c8f92790-kube-api-access-m22kb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7\" (UID: \"eea6e5c2-57f8-42c6-9ce9-55a3c8f92790\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7"
Nov 03 21:14:34.446527 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.446503    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-image-registry-private-configuration\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.446596 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.446547    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhdwv\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-kube-api-access-bhdwv\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.446596 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.446557    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 21:14:34.446596 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.446576    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-certificates\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.446724 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.446606    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eea6e5c2-57f8-42c6-9ce9-55a3c8f92790-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7\" (UID: \"eea6e5c2-57f8-42c6-9ce9-55a3c8f92790\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7"
Nov 03 21:14:34.446724 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.446628    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-networking-console-plugin-cert podName:bb8a800c-36be-4da0-b1e6-5ef65f994fc1 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:34.946609085 +0000 UTC m=+31.245235860 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j2ksl" (UID: "bb8a800c-36be-4da0-b1e6-5ef65f994fc1") : secret "networking-console-plugin-cert" not found
Nov 03 21:14:34.446724 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.446678    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ef7a859e-70f5-497a-9ad4-6614885fb71e-image-registry-private-configuration\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.446724 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.446709    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-bound-sa-token\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.446974 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.446737    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-config-volume\") pod \"dns-default-8p8hk\" (UID: \"aadb342f-3310-4c81-967d-0f0a8ef5a4e6\") " pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:34.446974 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.446764    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c27ff312-c8d3-4691-b535-ea8adf44d9c3-cert\") pod \"ingress-canary-m5s9b\" (UID: \"c27ff312-c8d3-4691-b535-ea8adf44d9c3\") " pod="openshift-ingress-canary/ingress-canary-m5s9b"
Nov 03 21:14:34.446974 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.446806    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-trusted-ca\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.446974 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.446837    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv8v4\" (UniqueName: \"kubernetes.io/projected/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-kube-api-access-pv8v4\") pod \"dns-default-8p8hk\" (UID: \"aadb342f-3310-4c81-967d-0f0a8ef5a4e6\") " pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:34.446974 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.446861    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ef7a859e-70f5-497a-9ad4-6614885fb71e-ca-trust-extracted\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.447408 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.447345    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-ca-trust-extracted\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.447408 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.446864    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-ca-trust-extracted\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.447571 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.447418    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpzsk\" (UniqueName: \"kubernetes.io/projected/c27ff312-c8d3-4691-b535-ea8adf44d9c3-kube-api-access-zpzsk\") pod \"ingress-canary-m5s9b\" (UID: \"c27ff312-c8d3-4691-b535-ea8adf44d9c3\") " pod="openshift-ingress-canary/ingress-canary-m5s9b"
Nov 03 21:14:34.447571 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.447450    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j2ksl\" (UID: \"bb8a800c-36be-4da0-b1e6-5ef65f994fc1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j2ksl"
Nov 03 21:14:34.447571 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.447487    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eea6e5c2-57f8-42c6-9ce9-55a3c8f92790-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7\" (UID: \"eea6e5c2-57f8-42c6-9ce9-55a3c8f92790\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7"
Nov 03 21:14:34.447571 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.447501    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.447571 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.447527    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jrqs\" (UniqueName: \"kubernetes.io/projected/06a672d7-0667-4bab-98ee-91dd71dfc489-kube-api-access-9jrqs\") pod \"network-check-source-6f78d59f99-sb5xl\" (UID: \"06a672d7-0667-4bab-98ee-91dd71dfc489\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sb5xl"
Nov 03 21:14:34.447987 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.447968    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 21:14:34.448043 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.447990    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75cb56744b-c6rsx: secret "image-registry-tls" not found
Nov 03 21:14:34.448043 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.448041    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls podName:ef7a859e-70f5-497a-9ad4-6614885fb71e nodeName:}" failed. No retries permitted until 2026-11-03 21:14:34.948023042 +0000 UTC m=+31.246649814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls") pod "image-registry-75cb56744b-c6rsx" (UID: "ef7a859e-70f5-497a-9ad4-6614885fb71e") : secret "image-registry-tls" not found
Nov 03 21:14:34.448153 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.448110    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-trusted-ca\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.448153 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.448121    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j2ksl\" (UID: \"bb8a800c-36be-4da0-b1e6-5ef65f994fc1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j2ksl"
Nov 03 21:14:34.448274 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.448182    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-certificates\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.448274 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.448208    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-metrics-tls\") pod \"dns-default-8p8hk\" (UID: \"aadb342f-3310-4c81-967d-0f0a8ef5a4e6\") " pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:34.448274 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.448259    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-tmp-dir\") pod \"dns-default-8p8hk\" (UID: \"aadb342f-3310-4c81-967d-0f0a8ef5a4e6\") " pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:34.448424 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.448289    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ef7a859e-70f5-497a-9ad4-6614885fb71e-installation-pull-secrets\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.448424 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.448322    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.448424 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.448351    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp68p\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-kube-api-access-lp68p\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.448424 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.448379    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef7a859e-70f5-497a-9ad4-6614885fb71e-trusted-ca\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.448590 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.448432    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eea6e5c2-57f8-42c6-9ce9-55a3c8f92790-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7\" (UID: \"eea6e5c2-57f8-42c6-9ce9-55a3c8f92790\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7"
Nov 03 21:14:34.448590 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.448463    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-bound-sa-token\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.448734 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.448711    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-certificates\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.448809 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.448792    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 21:14:34.448856 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.448814    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 21:14:34.448856 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.448827    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6df897dfcf-2b92c: secret "image-registry-tls" not found
Nov 03 21:14:34.448856 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.448846    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-metrics-tls podName:aadb342f-3310-4c81-967d-0f0a8ef5a4e6 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:34.948831613 +0000 UTC m=+31.247458379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-metrics-tls") pod "dns-default-8p8hk" (UID: "aadb342f-3310-4c81-967d-0f0a8ef5a4e6") : secret "dns-default-metrics-tls" not found
Nov 03 21:14:34.449007 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.448868    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls podName:1db07cc2-aa71-490e-a21a-0a8cb06d94d9 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:34.948853042 +0000 UTC m=+31.247479803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls") pod "image-registry-6df897dfcf-2b92c" (UID: "1db07cc2-aa71-490e-a21a-0a8cb06d94d9") : secret "image-registry-tls" not found
Nov 03 21:14:34.449007 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.448903    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-config-volume\") pod \"dns-default-8p8hk\" (UID: \"aadb342f-3310-4c81-967d-0f0a8ef5a4e6\") " pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:34.449125 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.449103    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-tmp-dir\") pod \"dns-default-8p8hk\" (UID: \"aadb342f-3310-4c81-967d-0f0a8ef5a4e6\") " pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:34.450121 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.449615    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-certificates\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.450121 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.449894    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef7a859e-70f5-497a-9ad4-6614885fb71e-trusted-ca\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.450121 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.449982    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ef7a859e-70f5-497a-9ad4-6614885fb71e-image-registry-private-configuration\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.450657 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.450616    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c27ff312-c8d3-4691-b535-ea8adf44d9c3-cert\") pod \"ingress-canary-m5s9b\" (UID: \"c27ff312-c8d3-4691-b535-ea8adf44d9c3\") " pod="openshift-ingress-canary/ingress-canary-m5s9b"
Nov 03 21:14:34.450883 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.450865    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-image-registry-private-configuration\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.450981 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.450956    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-installation-pull-secrets\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.451486 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.451468    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ef7a859e-70f5-497a-9ad4-6614885fb71e-installation-pull-secrets\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.451568 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.451546    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-27lr2"
Nov 03 21:14:34.451744 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.451726    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eea6e5c2-57f8-42c6-9ce9-55a3c8f92790-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7\" (UID: \"eea6e5c2-57f8-42c6-9ce9-55a3c8f92790\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7"
Nov 03 21:14:34.470220 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.470195    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bhpkz"
Nov 03 21:14:34.496698 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.496666    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:34.508116 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.508086    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nv9qq"
Nov 03 21:14:34.508700 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.508679    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m22kb\" (UniqueName: \"kubernetes.io/projected/eea6e5c2-57f8-42c6-9ce9-55a3c8f92790-kube-api-access-m22kb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7\" (UID: \"eea6e5c2-57f8-42c6-9ce9-55a3c8f92790\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7"
Nov 03 21:14:34.509427 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.509395    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhdwv\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-kube-api-access-bhdwv\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.518772 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.511559    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp68p\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-kube-api-access-lp68p\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.518772 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.513587    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jrqs\" (UniqueName: \"kubernetes.io/projected/06a672d7-0667-4bab-98ee-91dd71dfc489-kube-api-access-9jrqs\") pod \"network-check-source-6f78d59f99-sb5xl\" (UID: \"06a672d7-0667-4bab-98ee-91dd71dfc489\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sb5xl"
Nov 03 21:14:34.518772 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.518620    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv8v4\" (UniqueName: \"kubernetes.io/projected/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-kube-api-access-pv8v4\") pod \"dns-default-8p8hk\" (UID: \"aadb342f-3310-4c81-967d-0f0a8ef5a4e6\") " pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:34.519316 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.519263    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpzsk\" (UniqueName: \"kubernetes.io/projected/c27ff312-c8d3-4691-b535-ea8adf44d9c3-kube-api-access-zpzsk\") pod \"ingress-canary-m5s9b\" (UID: \"c27ff312-c8d3-4691-b535-ea8adf44d9c3\") " pod="openshift-ingress-canary/ingress-canary-m5s9b"
Nov 03 21:14:34.528345 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.528133    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:34.528907 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.528841    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-bound-sa-token\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.529329 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.529276    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-bound-sa-token\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.549726 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.549696    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8"
Nov 03 21:14:34.564608 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.564419    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7"
Nov 03 21:14:34.627417 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.627322    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sb5xl"
Nov 03 21:14:34.629772 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.629737    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m5s9b"
Nov 03 21:14:34.845734 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.845714    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-27lr2"]
Nov 03 21:14:34.848447 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.848424    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-sds64"]
Nov 03 21:14:34.848613 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:34.848594    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod974ed564_6386_4e1e_9326_f14ccc36658c.slice/crio-2283513bde82fc74d844a3598f31a3ab9f6a0e1b59d1aab361f8cadf549d543b WatchSource:0}: Error finding container 2283513bde82fc74d844a3598f31a3ab9f6a0e1b59d1aab361f8cadf549d543b: Status 404 returned error can't find the container with id 2283513bde82fc74d844a3598f31a3ab9f6a0e1b59d1aab361f8cadf549d543b
Nov 03 21:14:34.850804 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:34.850777    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52709fc1_b443_4a46_a95e_20cacc4ac4a0.slice/crio-39fa4173aebabde72c8d3f7f8b97eae39ea8e8495f4c98be0237f5a3f3e78f0c WatchSource:0}: Error finding container 39fa4173aebabde72c8d3f7f8b97eae39ea8e8495f4c98be0237f5a3f3e78f0c: Status 404 returned error can't find the container with id 39fa4173aebabde72c8d3f7f8b97eae39ea8e8495f4c98be0237f5a3f3e78f0c
Nov 03 21:14:34.851804 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.851784    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:34.851880 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.851842    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jf555\" (UID: \"ae204c86-8d93-4adf-988c-cd836e011270\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:34.852109 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.851934    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle podName:7ccca940-de12-451b-b920-d6bef516942f nodeName:}" failed. No retries permitted until 2026-11-03 21:14:35.851912238 +0000 UTC m=+32.150539025 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle") pod "router-default-5ffbbbfcf7-99qbj" (UID: "7ccca940-de12-451b-b920-d6bef516942f") : configmap references non-existent config key: service-ca.crt
Nov 03 21:14:34.852109 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.851986    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 21:14:34.852109 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.852032    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls podName:ae204c86-8d93-4adf-988c-cd836e011270 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:35.852020167 +0000 UTC m=+32.150646942 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jf555" (UID: "ae204c86-8d93-4adf-988c-cd836e011270") : secret "cluster-monitoring-operator-tls" not found
Nov 03 21:14:34.937121 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.936855    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-bc6rw"]
Nov 03 21:14:34.938992 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:34.938969    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cacf547_c691_4b14_936c_e6b08691bf93.slice/crio-aff104ed3eabbda7a26979c42d4f035cc30f6e99b4a2197a14e2014b3c91c1e2 WatchSource:0}: Error finding container aff104ed3eabbda7a26979c42d4f035cc30f6e99b4a2197a14e2014b3c91c1e2: Status 404 returned error can't find the container with id aff104ed3eabbda7a26979c42d4f035cc30f6e99b4a2197a14e2014b3c91c1e2
Nov 03 21:14:34.941545 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.941522    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nv9qq"]
Nov 03 21:14:34.944396 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:34.944360    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35650ff5_011e_4572_886f_8bf47f598219.slice/crio-7de20525aa734c869cf50481e66383dba843e1abc1346e0984414ac65ba8d0dc WatchSource:0}: Error finding container 7de20525aa734c869cf50481e66383dba843e1abc1346e0984414ac65ba8d0dc: Status 404 returned error can't find the container with id 7de20525aa734c869cf50481e66383dba843e1abc1346e0984414ac65ba8d0dc
Nov 03 21:14:34.952964 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.952944    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j2ksl\" (UID: \"bb8a800c-36be-4da0-b1e6-5ef65f994fc1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j2ksl"
Nov 03 21:14:34.953059 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.953015    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sqlqz\" (UID: \"cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"
Nov 03 21:14:34.953124 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.953055    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:34.953124 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.953101    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 21:14:34.953124 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.953111    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:34.953301 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.953151    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 21:14:34.953301 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.953153    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-metrics-tls\") pod \"dns-default-8p8hk\" (UID: \"aadb342f-3310-4c81-967d-0f0a8ef5a4e6\") " pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:34.953301 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.953198    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:34.953301 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.953206    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 21:14:34.953301 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.953213    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-networking-console-plugin-cert podName:bb8a800c-36be-4da0-b1e6-5ef65f994fc1 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:35.95319512 +0000 UTC m=+32.251821884 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j2ksl" (UID: "bb8a800c-36be-4da0-b1e6-5ef65f994fc1") : secret "networking-console-plugin-cert" not found
Nov 03 21:14:34.953301 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.953216    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75cb56744b-c6rsx: secret "image-registry-tls" not found
Nov 03 21:14:34.953301 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.953258    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 21:14:34.953301 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.953271    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 21:14:34.953636 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.953278    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-tls podName:cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c nodeName:}" failed. No retries permitted until 2026-11-03 21:14:35.953227515 +0000 UTC m=+32.251854280 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-sqlqz" (UID: "cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c") : secret "samples-operator-tls" not found
Nov 03 21:14:34.953636 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.953307    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 21:14:34.953636 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.953457    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6df897dfcf-2b92c: secret "image-registry-tls" not found
Nov 03 21:14:34.953636 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.953483    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls podName:ef7a859e-70f5-497a-9ad4-6614885fb71e nodeName:}" failed. No retries permitted until 2026-11-03 21:14:35.953456846 +0000 UTC m=+32.252083606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls") pod "image-registry-75cb56744b-c6rsx" (UID: "ef7a859e-70f5-497a-9ad4-6614885fb71e") : secret "image-registry-tls" not found
Nov 03 21:14:34.953636 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.953506    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-metrics-tls podName:aadb342f-3310-4c81-967d-0f0a8ef5a4e6 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:35.953498229 +0000 UTC m=+32.252125003 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-metrics-tls") pod "dns-default-8p8hk" (UID: "aadb342f-3310-4c81-967d-0f0a8ef5a4e6") : secret "dns-default-metrics-tls" not found
Nov 03 21:14:34.953636 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.953524    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs podName:7ccca940-de12-451b-b920-d6bef516942f nodeName:}" failed. No retries permitted until 2026-11-03 21:14:35.953518164 +0000 UTC m=+32.252144997 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs") pod "router-default-5ffbbbfcf7-99qbj" (UID: "7ccca940-de12-451b-b920-d6bef516942f") : secret "router-metrics-certs-default" not found
Nov 03 21:14:34.953636 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:34.953538    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls podName:1db07cc2-aa71-490e-a21a-0a8cb06d94d9 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:35.953533504 +0000 UTC m=+32.252160299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls") pod "image-registry-6df897dfcf-2b92c" (UID: "1db07cc2-aa71-490e-a21a-0a8cb06d94d9") : secret "image-registry-tls" not found
Nov 03 21:14:34.985562 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.985493    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8"]
Nov 03 21:14:34.987555 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:34.987531    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067bc9bb_5bc9_4ef5_8118_bd9a7d4048fa.slice/crio-a4f7a821463b65fd299a7fdd03ef6a8cadd5ddeb430028d1e512a8d8c69dc117 WatchSource:0}: Error finding container a4f7a821463b65fd299a7fdd03ef6a8cadd5ddeb430028d1e512a8d8c69dc117: Status 404 returned error can't find the container with id a4f7a821463b65fd299a7fdd03ef6a8cadd5ddeb430028d1e512a8d8c69dc117
Nov 03 21:14:34.988943 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.988922    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7"]
Nov 03 21:14:34.991510 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.991486    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"]
Nov 03 21:14:34.992183 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:34.992161    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea6e5c2_57f8_42c6_9ce9_55a3c8f92790.slice/crio-d5ccbc0b80c91de3303cbf85baaa8e4ef8c73804e35bd2fc88396217f79adeda WatchSource:0}: Error finding container d5ccbc0b80c91de3303cbf85baaa8e4ef8c73804e35bd2fc88396217f79adeda: Status 404 returned error can't find the container with id d5ccbc0b80c91de3303cbf85baaa8e4ef8c73804e35bd2fc88396217f79adeda
Nov 03 21:14:34.994379 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:34.994366    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bhpkz"]
Nov 03 21:14:34.994651 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:34.994633    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6547c71f_71a2_40c9_912f_9283b6bdb560.slice/crio-42ea59ba3e76bc7fd70fe4215d37b5fcaa705382189b846299d9f3659b476d02 WatchSource:0}: Error finding container 42ea59ba3e76bc7fd70fe4215d37b5fcaa705382189b846299d9f3659b476d02: Status 404 returned error can't find the container with id 42ea59ba3e76bc7fd70fe4215d37b5fcaa705382189b846299d9f3659b476d02
Nov 03 21:14:34.996136 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:34.996114    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd3c724_e7e5_4382_b681_adf7fd8f302b.slice/crio-2671cefed3cb069b2b6da7234d7d5f384a14b35dd5a2ccbe39248b441f34666b WatchSource:0}: Error finding container 2671cefed3cb069b2b6da7234d7d5f384a14b35dd5a2ccbe39248b441f34666b: Status 404 returned error can't find the container with id 2671cefed3cb069b2b6da7234d7d5f384a14b35dd5a2ccbe39248b441f34666b
Nov 03 21:14:35.051854 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:35.051822    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m5s9b"]
Nov 03 21:14:35.055789 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:35.055686    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27ff312_c8d3_4691_b535_ea8adf44d9c3.slice/crio-9620b8e282fcd830eb914c216fa909a06260597ea5734be008fcfc0fe17e10e1 WatchSource:0}: Error finding container 9620b8e282fcd830eb914c216fa909a06260597ea5734be008fcfc0fe17e10e1: Status 404 returned error can't find the container with id 9620b8e282fcd830eb914c216fa909a06260597ea5734be008fcfc0fe17e10e1
Nov 03 21:14:35.084548 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:35.084528    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-sb5xl"]
Nov 03 21:14:35.086685 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:35.086661    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a672d7_0667_4bab_98ee_91dd71dfc489.slice/crio-f7c95aa9c9172a1e1e8603fc2173aeef5ad3b11c79e3bcf8254e50b01236a132 WatchSource:0}: Error finding container f7c95aa9c9172a1e1e8603fc2173aeef5ad3b11c79e3bcf8254e50b01236a132: Status 404 returned error can't find the container with id f7c95aa9c9172a1e1e8603fc2173aeef5ad3b11c79e3bcf8254e50b01236a132
Nov 03 21:14:35.402372 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:35.402337    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8" event={"ID":"067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa","Type":"ContainerStarted","Data":"a4f7a821463b65fd299a7fdd03ef6a8cadd5ddeb430028d1e512a8d8c69dc117"}
Nov 03 21:14:35.403551 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:35.403528    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nv9qq" event={"ID":"35650ff5-011e-4572-886f-8bf47f598219","Type":"ContainerStarted","Data":"7de20525aa734c869cf50481e66383dba843e1abc1346e0984414ac65ba8d0dc"}
Nov 03 21:14:35.405012 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:35.404991    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-27lr2" event={"ID":"974ed564-6386-4e1e-9326-f14ccc36658c","Type":"ContainerStarted","Data":"fbdea8555ac6446bb1bf98e7f059ec1a26a019029d3bf2df25dcb8316c0f8a54"}
Nov 03 21:14:35.405090 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:35.405027    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-27lr2" event={"ID":"974ed564-6386-4e1e-9326-f14ccc36658c","Type":"ContainerStarted","Data":"2283513bde82fc74d844a3598f31a3ab9f6a0e1b59d1aab361f8cadf549d543b"}
Nov 03 21:14:35.406553 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:35.406532    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-sds64" event={"ID":"52709fc1-b443-4a46-a95e-20cacc4ac4a0","Type":"ContainerStarted","Data":"7c2a97c559b4a095f9f067bdc4edcd09cf1c683dbf39882f1404eab70656c178"}
Nov 03 21:14:35.406634 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:35.406575    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-sds64" event={"ID":"52709fc1-b443-4a46-a95e-20cacc4ac4a0","Type":"ContainerStarted","Data":"39fa4173aebabde72c8d3f7f8b97eae39ea8e8495f4c98be0237f5a3f3e78f0c"}
Nov 03 21:14:35.407740 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:35.407711    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7" event={"ID":"eea6e5c2-57f8-42c6-9ce9-55a3c8f92790","Type":"ContainerStarted","Data":"d5ccbc0b80c91de3303cbf85baaa8e4ef8c73804e35bd2fc88396217f79adeda"}
Nov 03 21:14:35.409218 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:35.409201    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv" event={"ID":"6547c71f-71a2-40c9-912f-9283b6bdb560","Type":"ContainerStarted","Data":"42ea59ba3e76bc7fd70fe4215d37b5fcaa705382189b846299d9f3659b476d02"}
Nov 03 21:14:35.410451 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:35.410434    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-bc6rw" event={"ID":"5cacf547-c691-4b14-936c-e6b08691bf93","Type":"ContainerStarted","Data":"aff104ed3eabbda7a26979c42d4f035cc30f6e99b4a2197a14e2014b3c91c1e2"}
Nov 03 21:14:35.411848 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:35.411828    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bhpkz" event={"ID":"bbd3c724-e7e5-4382-b681-adf7fd8f302b","Type":"ContainerStarted","Data":"e0d8b7c28e00bd69cbd6ebdd2173228f71213ba7be0527eb4e2d0054ae29f7d8"}
Nov 03 21:14:35.411929 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:35.411865    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bhpkz" event={"ID":"bbd3c724-e7e5-4382-b681-adf7fd8f302b","Type":"ContainerStarted","Data":"2671cefed3cb069b2b6da7234d7d5f384a14b35dd5a2ccbe39248b441f34666b"}
Nov 03 21:14:35.413060 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:35.413042    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m5s9b" event={"ID":"c27ff312-c8d3-4691-b535-ea8adf44d9c3","Type":"ContainerStarted","Data":"9620b8e282fcd830eb914c216fa909a06260597ea5734be008fcfc0fe17e10e1"}
Nov 03 21:14:35.414172 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:35.414156    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sb5xl" event={"ID":"06a672d7-0667-4bab-98ee-91dd71dfc489","Type":"ContainerStarted","Data":"f7c95aa9c9172a1e1e8603fc2173aeef5ad3b11c79e3bcf8254e50b01236a132"}
Nov 03 21:14:35.861572 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:35.861539    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:35.861753 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:35.861601    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jf555\" (UID: \"ae204c86-8d93-4adf-988c-cd836e011270\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:35.861753 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:35.861728    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle podName:7ccca940-de12-451b-b920-d6bef516942f nodeName:}" failed. No retries permitted until 2026-11-03 21:14:37.861705084 +0000 UTC m=+34.160331863 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle") pod "router-default-5ffbbbfcf7-99qbj" (UID: "7ccca940-de12-451b-b920-d6bef516942f") : configmap references non-existent config key: service-ca.crt
Nov 03 21:14:35.861865 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:35.861784    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 21:14:35.861865 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:35.861843    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls podName:ae204c86-8d93-4adf-988c-cd836e011270 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:37.86182818 +0000 UTC m=+34.160454940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jf555" (UID: "ae204c86-8d93-4adf-988c-cd836e011270") : secret "cluster-monitoring-operator-tls" not found
Nov 03 21:14:35.962428 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:35.962395    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sqlqz\" (UID: \"cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"
Nov 03 21:14:35.962592 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:35.962448    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:35.962592 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:35.962504    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:35.962592 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:35.962541    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-metrics-tls\") pod \"dns-default-8p8hk\" (UID: \"aadb342f-3310-4c81-967d-0f0a8ef5a4e6\") " pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:35.962592 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:35.962574    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:35.962785 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:35.962647    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j2ksl\" (UID: \"bb8a800c-36be-4da0-b1e6-5ef65f994fc1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j2ksl"
Nov 03 21:14:35.962785 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:35.962773    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 21:14:35.962865 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:35.962831    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-networking-console-plugin-cert podName:bb8a800c-36be-4da0-b1e6-5ef65f994fc1 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:37.962814384 +0000 UTC m=+34.261441165 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j2ksl" (UID: "bb8a800c-36be-4da0-b1e6-5ef65f994fc1") : secret "networking-console-plugin-cert" not found
Nov 03 21:14:35.962905 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:35.962893    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 21:14:35.962937 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:35.962925    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-tls podName:cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c nodeName:}" failed. No retries permitted until 2026-11-03 21:14:37.962914878 +0000 UTC m=+34.261541650 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-sqlqz" (UID: "cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c") : secret "samples-operator-tls" not found
Nov 03 21:14:35.962981 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:35.962968    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 21:14:35.963013 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:35.962997    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs podName:7ccca940-de12-451b-b920-d6bef516942f nodeName:}" failed. No retries permitted until 2026-11-03 21:14:37.962987851 +0000 UTC m=+34.261614626 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs") pod "router-default-5ffbbbfcf7-99qbj" (UID: "7ccca940-de12-451b-b920-d6bef516942f") : secret "router-metrics-certs-default" not found
Nov 03 21:14:35.963047 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:35.963039    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 21:14:35.963077 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:35.963046    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75cb56744b-c6rsx: secret "image-registry-tls" not found
Nov 03 21:14:35.963077 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:35.963065    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls podName:ef7a859e-70f5-497a-9ad4-6614885fb71e nodeName:}" failed. No retries permitted until 2026-11-03 21:14:37.963059143 +0000 UTC m=+34.261685901 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls") pod "image-registry-75cb56744b-c6rsx" (UID: "ef7a859e-70f5-497a-9ad4-6614885fb71e") : secret "image-registry-tls" not found
Nov 03 21:14:35.963148 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:35.963100    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 21:14:35.963148 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:35.963116    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-metrics-tls podName:aadb342f-3310-4c81-967d-0f0a8ef5a4e6 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:37.963110949 +0000 UTC m=+34.261737706 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-metrics-tls") pod "dns-default-8p8hk" (UID: "aadb342f-3310-4c81-967d-0f0a8ef5a4e6") : secret "dns-default-metrics-tls" not found
Nov 03 21:14:35.963208 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:35.963167    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 21:14:35.963208 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:35.963174    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6df897dfcf-2b92c: secret "image-registry-tls" not found
Nov 03 21:14:35.963208 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:35.963191    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls podName:1db07cc2-aa71-490e-a21a-0a8cb06d94d9 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:37.963185932 +0000 UTC m=+34.261812689 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls") pod "image-registry-6df897dfcf-2b92c" (UID: "1db07cc2-aa71-490e-a21a-0a8cb06d94d9") : secret "image-registry-tls" not found
Nov 03 21:14:36.354922 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:36.354870    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-sds64" podStartSLOduration=436.354851396 podStartE2EDuration="7m16.354851396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:36.353992436 +0000 UTC m=+32.652619218" watchObservedRunningTime="2026-11-03 21:14:36.354851396 +0000 UTC m=+32.653478177"
Nov 03 21:14:36.469305 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:36.468726    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-27lr2" podStartSLOduration=436.468706357 podStartE2EDuration="7m16.468706357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:36.467896385 +0000 UTC m=+32.766523167" watchObservedRunningTime="2026-11-03 21:14:36.468706357 +0000 UTC m=+32.767333139"
Nov 03 21:14:36.598792 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:36.598725    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-bhpkz" podStartSLOduration=436.598703612 podStartE2EDuration="7m16.598703612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:36.581619312 +0000 UTC m=+32.880246092" watchObservedRunningTime="2026-11-03 21:14:36.598703612 +0000 UTC m=+32.897330436"
Nov 03 21:14:37.174411 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:37.174377    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-original-pull-secret\") pod \"global-pull-secret-syncer-v5vrf\" (UID: \"96fee653-58b4-46bf-9ed5-0d3c5a93e8a8\") " pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:37.174411 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:37.174426    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs\") pod \"network-metrics-daemon-6xvgt\" (UID: \"4b84ba29-a116-48ef-8d9f-d81ceb4b3359\") " pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:37.174748 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:37.174675    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 21:14:37.174748 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:37.174741    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs podName:4b84ba29-a116-48ef-8d9f-d81ceb4b3359 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:09.17472179 +0000 UTC m=+65.473348563 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs") pod "network-metrics-daemon-6xvgt" (UID: "4b84ba29-a116-48ef-8d9f-d81ceb4b3359") : secret "metrics-daemon-secret" not found
Nov 03 21:14:37.188931 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:37.188847    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-original-pull-secret\") pod \"global-pull-secret-syncer-v5vrf\" (UID: \"96fee653-58b4-46bf-9ed5-0d3c5a93e8a8\") " pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:37.214591 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:37.214565    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:37.275829 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:37.275252    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb9kd\" (UniqueName: \"kubernetes.io/projected/784231bb-189f-472c-83ed-6ced092f10a1-kube-api-access-nb9kd\") pod \"network-check-target-8w5st\" (UID: \"784231bb-189f-472c-83ed-6ced092f10a1\") " pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:37.280727 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:37.280698    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb9kd\" (UniqueName: \"kubernetes.io/projected/784231bb-189f-472c-83ed-6ced092f10a1-kube-api-access-nb9kd\") pod \"network-check-target-8w5st\" (UID: \"784231bb-189f-472c-83ed-6ced092f10a1\") " pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:37.426663 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:37.426561    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nv9qq" event={"ID":"35650ff5-011e-4572-886f-8bf47f598219","Type":"ContainerStarted","Data":"e00c8865e7b4ef293889823cb511eb26014243c49e5c63af95eb6f90423f0719"}
Nov 03 21:14:37.521005 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:37.520975    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:37.542231 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:37.542172    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nv9qq" podStartSLOduration=411.238017485 podStartE2EDuration="6m53.542149755s" totalImagesPullingTime="2.30413227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:37.540397362 +0000 UTC m=+33.839024154" watchObservedRunningTime="2026-11-03 21:14:37.542149755 +0000 UTC m=+33.840776537"
Nov 03 21:14:37.633291 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:37.633229    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v5vrf"]
Nov 03 21:14:37.881945 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:37.881900    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:37.882137 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:37.881987    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jf555\" (UID: \"ae204c86-8d93-4adf-988c-cd836e011270\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:37.882137 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:37.882094    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle podName:7ccca940-de12-451b-b920-d6bef516942f nodeName:}" failed. No retries permitted until 2026-11-03 21:14:41.882070278 +0000 UTC m=+38.180697051 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle") pod "router-default-5ffbbbfcf7-99qbj" (UID: "7ccca940-de12-451b-b920-d6bef516942f") : configmap references non-existent config key: service-ca.crt
Nov 03 21:14:37.882281 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:37.882167    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 21:14:37.882281 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:37.882224    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls podName:ae204c86-8d93-4adf-988c-cd836e011270 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:41.882211183 +0000 UTC m=+38.180837959 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jf555" (UID: "ae204c86-8d93-4adf-988c-cd836e011270") : secret "cluster-monitoring-operator-tls" not found
Nov 03 21:14:37.944030 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:37.943531    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8w5st"]
Nov 03 21:14:37.950148 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:37.950110    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784231bb_189f_472c_83ed_6ced092f10a1.slice/crio-d3bc865020f000a0564451e1037d99663f52b28218d998575b70fa808077b428 WatchSource:0}: Error finding container d3bc865020f000a0564451e1037d99663f52b28218d998575b70fa808077b428: Status 404 returned error can't find the container with id d3bc865020f000a0564451e1037d99663f52b28218d998575b70fa808077b428
Nov 03 21:14:37.982809 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:37.982770    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j2ksl\" (UID: \"bb8a800c-36be-4da0-b1e6-5ef65f994fc1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j2ksl"
Nov 03 21:14:37.982960 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:37.982863    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sqlqz\" (UID: \"cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"
Nov 03 21:14:37.982960 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:37.982901    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:37.982960 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:37.982950    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:37.983097 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:37.982989    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-metrics-tls\") pod \"dns-default-8p8hk\" (UID: \"aadb342f-3310-4c81-967d-0f0a8ef5a4e6\") " pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:37.983097 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:37.983026    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:37.983208 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:37.983189    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 21:14:37.983278 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:37.983211    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6df897dfcf-2b92c: secret "image-registry-tls" not found
Nov 03 21:14:37.983338 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:37.983287    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls podName:1db07cc2-aa71-490e-a21a-0a8cb06d94d9 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:41.98326906 +0000 UTC m=+38.281895837 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls") pod "image-registry-6df897dfcf-2b92c" (UID: "1db07cc2-aa71-490e-a21a-0a8cb06d94d9") : secret "image-registry-tls" not found
Nov 03 21:14:37.983406 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:37.983361    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 21:14:37.983406 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:37.983398    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-networking-console-plugin-cert podName:bb8a800c-36be-4da0-b1e6-5ef65f994fc1 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:41.983386871 +0000 UTC m=+38.282013631 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j2ksl" (UID: "bb8a800c-36be-4da0-b1e6-5ef65f994fc1") : secret "networking-console-plugin-cert" not found
Nov 03 21:14:37.983512 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:37.983460    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 21:14:37.983512 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:37.983495    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-tls podName:cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c nodeName:}" failed. No retries permitted until 2026-11-03 21:14:41.98348369 +0000 UTC m=+38.282110463 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-sqlqz" (UID: "cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c") : secret "samples-operator-tls" not found
Nov 03 21:14:37.983615 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:37.983552    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 21:14:37.983615 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:37.983579    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs podName:7ccca940-de12-451b-b920-d6bef516942f nodeName:}" failed. No retries permitted until 2026-11-03 21:14:41.983569926 +0000 UTC m=+38.282196697 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs") pod "router-default-5ffbbbfcf7-99qbj" (UID: "7ccca940-de12-451b-b920-d6bef516942f") : secret "router-metrics-certs-default" not found
Nov 03 21:14:37.983722 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:37.983645    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 21:14:37.983722 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:37.983654    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75cb56744b-c6rsx: secret "image-registry-tls" not found
Nov 03 21:14:37.983722 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:37.983681    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls podName:ef7a859e-70f5-497a-9ad4-6614885fb71e nodeName:}" failed. No retries permitted until 2026-11-03 21:14:41.983672329 +0000 UTC m=+38.282299090 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls") pod "image-registry-75cb56744b-c6rsx" (UID: "ef7a859e-70f5-497a-9ad4-6614885fb71e") : secret "image-registry-tls" not found
Nov 03 21:14:37.983865 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:37.983751    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 21:14:37.983865 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:37.983780    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-metrics-tls podName:aadb342f-3310-4c81-967d-0f0a8ef5a4e6 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:41.983770896 +0000 UTC m=+38.282397658 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-metrics-tls") pod "dns-default-8p8hk" (UID: "aadb342f-3310-4c81-967d-0f0a8ef5a4e6") : secret "dns-default-metrics-tls" not found
Nov 03 21:14:38.435715 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:38.435674    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v5vrf" event={"ID":"96fee653-58b4-46bf-9ed5-0d3c5a93e8a8","Type":"ContainerStarted","Data":"e21bfbc1e9afbf9cd3a266677b0bd6c791030049a7071b4b81631715f0425b1d"}
Nov 03 21:14:38.439285 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:38.439257    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8w5st" event={"ID":"784231bb-189f-472c-83ed-6ced092f10a1","Type":"ContainerStarted","Data":"d3bc865020f000a0564451e1037d99663f52b28218d998575b70fa808077b428"}
Nov 03 21:14:40.449849 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:40.449818    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/0.log"
Nov 03 21:14:40.450297 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:40.449874    2574 generic.go:356] "Generic (PLEG): container finished" podID="6547c71f-71a2-40c9-912f-9283b6bdb560" containerID="cd447bddc8d65b26d8a66f13c52cbcb737972b3cfd1cd94c84c2ce2e1608f051" exitCode=255
Nov 03 21:14:40.450297 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:40.449949    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv" event={"ID":"6547c71f-71a2-40c9-912f-9283b6bdb560","Type":"ContainerDied","Data":"cd447bddc8d65b26d8a66f13c52cbcb737972b3cfd1cd94c84c2ce2e1608f051"}
Nov 03 21:14:40.450297 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:40.450174    2574 scope.go:122] "RemoveContainer" containerID="cd447bddc8d65b26d8a66f13c52cbcb737972b3cfd1cd94c84c2ce2e1608f051"
Nov 03 21:14:40.453973 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:40.453942    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-bc6rw" event={"ID":"5cacf547-c691-4b14-936c-e6b08691bf93","Type":"ContainerStarted","Data":"5a02761cdb34a1806e2c1620bf7eff9ff6f6f8477be95f582877726937c31d50"}
Nov 03 21:14:40.688677 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:40.687802    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-bc6rw" podStartSLOduration=411.684668869 podStartE2EDuration="6m56.687786302s" totalImagesPullingTime="5.003117433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:40.686532823 +0000 UTC m=+36.985159604" watchObservedRunningTime="2026-11-03 21:14:40.687786302 +0000 UTC m=+36.986413081"
Nov 03 21:14:41.341962 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.340835    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-v9vxl"]
Nov 03 21:14:41.356921 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.356886    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:41.401412 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.401378    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v9vxl"]
Nov 03 21:14:41.410179 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.410157    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 21:14:41.410339 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.410224    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 21:14:41.450674 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.450609    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tq7pf\""
Nov 03 21:14:41.469352 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.469318    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8" event={"ID":"067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa","Type":"ContainerStarted","Data":"48f1317b67a9f3effa8ab84804bda042214866a8cd3e7213d9e7ab044f795c78"}
Nov 03 21:14:41.475784 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.475752    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7" event={"ID":"eea6e5c2-57f8-42c6-9ce9-55a3c8f92790","Type":"ContainerStarted","Data":"67117625ad30e140b253bdd2bc78373b0e1011f190d089c545d45d2c350f6500"}
Nov 03 21:14:41.477943 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.477921    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/1.log"
Nov 03 21:14:41.478800 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.478779    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/0.log"
Nov 03 21:14:41.478900 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.478823    2574 generic.go:356] "Generic (PLEG): container finished" podID="6547c71f-71a2-40c9-912f-9283b6bdb560" containerID="9f6b6dc724c5973a7c58dc058c00d7b81472be8fbb45b2fedb085ed90d0ec098" exitCode=255
Nov 03 21:14:41.478964 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.478914    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv" event={"ID":"6547c71f-71a2-40c9-912f-9283b6bdb560","Type":"ContainerDied","Data":"9f6b6dc724c5973a7c58dc058c00d7b81472be8fbb45b2fedb085ed90d0ec098"}
Nov 03 21:14:41.479020 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.478972    2574 scope.go:122] "RemoveContainer" containerID="cd447bddc8d65b26d8a66f13c52cbcb737972b3cfd1cd94c84c2ce2e1608f051"
Nov 03 21:14:41.479145 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.479127    2574 scope.go:122] "RemoveContainer" containerID="9f6b6dc724c5973a7c58dc058c00d7b81472be8fbb45b2fedb085ed90d0ec098"
Nov 03 21:14:41.479403 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:41.479379    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rhpjv_openshift-console-operator(6547c71f-71a2-40c9-912f-9283b6bdb560)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv" podUID="6547c71f-71a2-40c9-912f-9283b6bdb560"
Nov 03 21:14:41.534729 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.534696    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cad3df5d-3fe9-449b-8b70-793bceb47985-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:41.534729 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.534734    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cad3df5d-3fe9-449b-8b70-793bceb47985-data-volume\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:41.534955 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.534817    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc77l\" (UniqueName: \"kubernetes.io/projected/cad3df5d-3fe9-449b-8b70-793bceb47985-kube-api-access-wc77l\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:41.534955 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.534930    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cad3df5d-3fe9-449b-8b70-793bceb47985-crio-socket\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:41.535070 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.535016    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:41.605681 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.605614    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8" podStartSLOduration=411.825183531 podStartE2EDuration="6m57.605594862s" totalImagesPullingTime="5.780411331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:41.592296809 +0000 UTC m=+37.890923590" watchObservedRunningTime="2026-11-03 21:14:41.605594862 +0000 UTC m=+37.904221643"
Nov 03 21:14:41.606002 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.605985    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-g82mg-mqwqf"]
Nov 03 21:14:41.634922 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.631687    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-g82mg-mqwqf"
Nov 03 21:14:41.637107 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.635623    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cad3df5d-3fe9-449b-8b70-793bceb47985-crio-socket\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:41.637107 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.635697    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:41.637107 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.635838    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cad3df5d-3fe9-449b-8b70-793bceb47985-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:41.637107 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.635875    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cad3df5d-3fe9-449b-8b70-793bceb47985-data-volume\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:41.637107 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.635932    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc77l\" (UniqueName: \"kubernetes.io/projected/cad3df5d-3fe9-449b-8b70-793bceb47985-kube-api-access-wc77l\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:41.637107 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:41.636366    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 21:14:41.637107 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:41.636433    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls podName:cad3df5d-3fe9-449b-8b70-793bceb47985 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:42.136413305 +0000 UTC m=+38.435040094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9vxl" (UID: "cad3df5d-3fe9-449b-8b70-793bceb47985") : secret "insights-runtime-extractor-tls" not found
Nov 03 21:14:41.637107 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:41.636946    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 21:14:41.637107 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:41.637011    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cad3df5d-3fe9-449b-8b70-793bceb47985-kube-rbac-proxy-cm podName:cad3df5d-3fe9-449b-8b70-793bceb47985 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:42.136993551 +0000 UTC m=+38.435620324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cad3df5d-3fe9-449b-8b70-793bceb47985-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v9vxl" (UID: "cad3df5d-3fe9-449b-8b70-793bceb47985") : configmap "kube-rbac-proxy" not found
Nov 03 21:14:41.637671 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.637162    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cad3df5d-3fe9-449b-8b70-793bceb47985-data-volume\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:41.637671 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.637422    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cad3df5d-3fe9-449b-8b70-793bceb47985-crio-socket\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:41.663564 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.663532    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-g82mg-mqwqf"]
Nov 03 21:14:41.685995 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.685963    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-md6qp\""
Nov 03 21:14:41.701283 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.701202    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc77l\" (UniqueName: \"kubernetes.io/projected/cad3df5d-3fe9-449b-8b70-793bceb47985-kube-api-access-wc77l\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:41.737338 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.737275    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/caaa9f42-f8a2-4459-8917-a53e8fd852cc-archives-path\") pod \"periodic-gathering-g82mg-mqwqf\" (UID: \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\") " pod="openshift-insights/periodic-gathering-g82mg-mqwqf"
Nov 03 21:14:41.737521 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.737386    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caaa9f42-f8a2-4459-8917-a53e8fd852cc-trusted-ca-bundle\") pod \"periodic-gathering-g82mg-mqwqf\" (UID: \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\") " pod="openshift-insights/periodic-gathering-g82mg-mqwqf"
Nov 03 21:14:41.737521 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.737492    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mddbv\" (UniqueName: \"kubernetes.io/projected/caaa9f42-f8a2-4459-8917-a53e8fd852cc-kube-api-access-mddbv\") pod \"periodic-gathering-g82mg-mqwqf\" (UID: \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\") " pod="openshift-insights/periodic-gathering-g82mg-mqwqf"
Nov 03 21:14:41.737709 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.737633    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caaa9f42-f8a2-4459-8917-a53e8fd852cc-service-ca-bundle\") pod \"periodic-gathering-g82mg-mqwqf\" (UID: \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\") " pod="openshift-insights/periodic-gathering-g82mg-mqwqf"
Nov 03 21:14:41.826866 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.826754    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7" podStartSLOduration=412.430787532 podStartE2EDuration="6m57.826735258s" totalImagesPullingTime="5.395947726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:41.825604624 +0000 UTC m=+38.124231417" watchObservedRunningTime="2026-11-03 21:14:41.826735258 +0000 UTC m=+38.125362040"
Nov 03 21:14:41.838735 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.838698    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/caaa9f42-f8a2-4459-8917-a53e8fd852cc-archives-path\") pod \"periodic-gathering-g82mg-mqwqf\" (UID: \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\") " pod="openshift-insights/periodic-gathering-g82mg-mqwqf"
Nov 03 21:14:41.838911 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.838746    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caaa9f42-f8a2-4459-8917-a53e8fd852cc-trusted-ca-bundle\") pod \"periodic-gathering-g82mg-mqwqf\" (UID: \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\") " pod="openshift-insights/periodic-gathering-g82mg-mqwqf"
Nov 03 21:14:41.838911 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.838816    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mddbv\" (UniqueName: \"kubernetes.io/projected/caaa9f42-f8a2-4459-8917-a53e8fd852cc-kube-api-access-mddbv\") pod \"periodic-gathering-g82mg-mqwqf\" (UID: \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\") " pod="openshift-insights/periodic-gathering-g82mg-mqwqf"
Nov 03 21:14:41.839036 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.838917    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caaa9f42-f8a2-4459-8917-a53e8fd852cc-service-ca-bundle\") pod \"periodic-gathering-g82mg-mqwqf\" (UID: \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\") " pod="openshift-insights/periodic-gathering-g82mg-mqwqf"
Nov 03 21:14:41.839542 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.839520    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caaa9f42-f8a2-4459-8917-a53e8fd852cc-service-ca-bundle\") pod \"periodic-gathering-g82mg-mqwqf\" (UID: \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\") " pod="openshift-insights/periodic-gathering-g82mg-mqwqf"
Nov 03 21:14:41.839785 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.839767    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/caaa9f42-f8a2-4459-8917-a53e8fd852cc-archives-path\") pod \"periodic-gathering-g82mg-mqwqf\" (UID: \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\") " pod="openshift-insights/periodic-gathering-g82mg-mqwqf"
Nov 03 21:14:41.840493 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.840469    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caaa9f42-f8a2-4459-8917-a53e8fd852cc-trusted-ca-bundle\") pod \"periodic-gathering-g82mg-mqwqf\" (UID: \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\") " pod="openshift-insights/periodic-gathering-g82mg-mqwqf"
Nov 03 21:14:41.898514 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.898480    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mddbv\" (UniqueName: \"kubernetes.io/projected/caaa9f42-f8a2-4459-8917-a53e8fd852cc-kube-api-access-mddbv\") pod \"periodic-gathering-g82mg-mqwqf\" (UID: \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\") " pod="openshift-insights/periodic-gathering-g82mg-mqwqf"
Nov 03 21:14:41.939696 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.939658    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:41.939878 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.939730    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jf555\" (UID: \"ae204c86-8d93-4adf-988c-cd836e011270\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:41.939938 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:41.939899    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 21:14:41.939984 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:41.939969    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls podName:ae204c86-8d93-4adf-988c-cd836e011270 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:49.93995054 +0000 UTC m=+46.238577311 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jf555" (UID: "ae204c86-8d93-4adf-988c-cd836e011270") : secret "cluster-monitoring-operator-tls" not found
Nov 03 21:14:41.940053 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:41.940042    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle podName:7ccca940-de12-451b-b920-d6bef516942f nodeName:}" failed. No retries permitted until 2026-11-03 21:14:49.940032595 +0000 UTC m=+46.238659369 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle") pod "router-default-5ffbbbfcf7-99qbj" (UID: "7ccca940-de12-451b-b920-d6bef516942f") : configmap references non-existent config key: service-ca.crt
Nov 03 21:14:41.966623 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:41.966594    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-g82mg-mqwqf"
Nov 03 21:14:42.041466 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.041426    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:42.041597 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.041521    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-metrics-tls\") pod \"dns-default-8p8hk\" (UID: \"aadb342f-3310-4c81-967d-0f0a8ef5a4e6\") " pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:42.041597 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.041575    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:42.041709 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.041648    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j2ksl\" (UID: \"bb8a800c-36be-4da0-b1e6-5ef65f994fc1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j2ksl"
Nov 03 21:14:42.041759 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.041734    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sqlqz\" (UID: \"cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"
Nov 03 21:14:42.041811 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.041773    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:42.041923 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:42.041907    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 21:14:42.041983 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:42.041972    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs podName:7ccca940-de12-451b-b920-d6bef516942f nodeName:}" failed. No retries permitted until 2026-11-03 21:14:50.041955222 +0000 UTC m=+46.340581996 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs") pod "router-default-5ffbbbfcf7-99qbj" (UID: "7ccca940-de12-451b-b920-d6bef516942f") : secret "router-metrics-certs-default" not found
Nov 03 21:14:42.042064 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:42.042052    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 21:14:42.042116 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:42.042068    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75cb56744b-c6rsx: secret "image-registry-tls" not found
Nov 03 21:14:42.042116 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:42.042104    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls podName:ef7a859e-70f5-497a-9ad4-6614885fb71e nodeName:}" failed. No retries permitted until 2026-11-03 21:14:50.042092875 +0000 UTC m=+46.340719646 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls") pod "image-registry-75cb56744b-c6rsx" (UID: "ef7a859e-70f5-497a-9ad4-6614885fb71e") : secret "image-registry-tls" not found
Nov 03 21:14:42.042223 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:42.042161    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 21:14:42.042223 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:42.042193    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-metrics-tls podName:aadb342f-3310-4c81-967d-0f0a8ef5a4e6 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:50.042182883 +0000 UTC m=+46.340809655 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-metrics-tls") pod "dns-default-8p8hk" (UID: "aadb342f-3310-4c81-967d-0f0a8ef5a4e6") : secret "dns-default-metrics-tls" not found
Nov 03 21:14:42.042356 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:42.042288    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 21:14:42.042356 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:42.042298    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6df897dfcf-2b92c: secret "image-registry-tls" not found
Nov 03 21:14:42.042356 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:42.042327    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls podName:1db07cc2-aa71-490e-a21a-0a8cb06d94d9 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:50.042317549 +0000 UTC m=+46.340944322 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls") pod "image-registry-6df897dfcf-2b92c" (UID: "1db07cc2-aa71-490e-a21a-0a8cb06d94d9") : secret "image-registry-tls" not found
Nov 03 21:14:42.042493 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:42.042428    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 21:14:42.042493 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:42.042463    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-networking-console-plugin-cert podName:bb8a800c-36be-4da0-b1e6-5ef65f994fc1 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:50.042452863 +0000 UTC m=+46.341079636 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j2ksl" (UID: "bb8a800c-36be-4da0-b1e6-5ef65f994fc1") : secret "networking-console-plugin-cert" not found
Nov 03 21:14:42.042651 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:42.042497    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 21:14:42.042651 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:42.042575    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-tls podName:cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c nodeName:}" failed. No retries permitted until 2026-11-03 21:14:50.042556574 +0000 UTC m=+46.341183349 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-sqlqz" (UID: "cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c") : secret "samples-operator-tls" not found
Nov 03 21:14:42.142658 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.142287    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:42.142658 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:42.142424    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 21:14:42.142658 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.142443    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cad3df5d-3fe9-449b-8b70-793bceb47985-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:42.142658 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:42.142494    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls podName:cad3df5d-3fe9-449b-8b70-793bceb47985 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:43.14247717 +0000 UTC m=+39.441103952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9vxl" (UID: "cad3df5d-3fe9-449b-8b70-793bceb47985") : secret "insights-runtime-extractor-tls" not found
Nov 03 21:14:42.142658 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:42.142540    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 21:14:42.142658 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:42.142592    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cad3df5d-3fe9-449b-8b70-793bceb47985-kube-rbac-proxy-cm podName:cad3df5d-3fe9-449b-8b70-793bceb47985 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:43.142576039 +0000 UTC m=+39.441202811 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cad3df5d-3fe9-449b-8b70-793bceb47985-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v9vxl" (UID: "cad3df5d-3fe9-449b-8b70-793bceb47985") : configmap "kube-rbac-proxy" not found
Nov 03 21:14:42.346590 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.346550    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-g82mg-mqwqf"]
Nov 03 21:14:42.367976 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.367946    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-v6qdv"]
Nov 03 21:14:42.394346 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.393266    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jqkbh"]
Nov 03 21:14:42.394346 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.393366    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v6qdv"
Nov 03 21:14:42.405705 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.405679    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jqkbh"
Nov 03 21:14:42.427580 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.427499    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-v6qdv"]
Nov 03 21:14:42.435976 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.435951    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jqkbh"]
Nov 03 21:14:42.447032 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.447012    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 21:14:42.447383 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.447048    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 21:14:42.447383 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.447091    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8zbhm\""
Nov 03 21:14:42.487083 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.487048    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8w5st" event={"ID":"784231bb-189f-472c-83ed-6ced092f10a1","Type":"ContainerStarted","Data":"d7eed296706ee11eab7768d09b9bf6cb85dca10fc729312adfbd38e99d862605"}
Nov 03 21:14:42.488513 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.488426    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:42.495526 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.495497    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-g82mg-mqwqf" event={"ID":"caaa9f42-f8a2-4459-8917-a53e8fd852cc","Type":"ContainerStarted","Data":"6ae6898c171f21f23843d81effbaf5336ef32b69a67d3c34a5e14d14e6a3a3d2"}
Nov 03 21:14:42.495631 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.495564    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-g82mg-mqwqf" event={"ID":"caaa9f42-f8a2-4459-8917-a53e8fd852cc","Type":"ContainerStarted","Data":"e6af7c60346a713e8a6887b47105e85cf45efcebfd0f257edcc8cdf6bff15154"}
Nov 03 21:14:42.498399 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.498375    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/1.log"
Nov 03 21:14:42.499057 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.499037    2574 scope.go:122] "RemoveContainer" containerID="9f6b6dc724c5973a7c58dc058c00d7b81472be8fbb45b2fedb085ed90d0ec098"
Nov 03 21:14:42.499322 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:42.499297    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rhpjv_openshift-console-operator(6547c71f-71a2-40c9-912f-9283b6bdb560)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv" podUID="6547c71f-71a2-40c9-912f-9283b6bdb560"
Nov 03 21:14:42.504528 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.504503    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m5s9b" event={"ID":"c27ff312-c8d3-4691-b535-ea8adf44d9c3","Type":"ContainerStarted","Data":"289444dd7f1d9bff3a0eaf1d64acabfcfa019e59ace3ea606903c22c551ece74"}
Nov 03 21:14:42.547091 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.547056    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jchm\" (UniqueName: \"kubernetes.io/projected/2000161a-2aa8-4638-a4f7-fbe305e929ae-kube-api-access-9jchm\") pod \"migrator-66df5d496b-jqkbh\" (UID: \"2000161a-2aa8-4638-a4f7-fbe305e929ae\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jqkbh"
Nov 03 21:14:42.547194 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.547177    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxb9x\" (UniqueName: \"kubernetes.io/projected/356632db-499f-4ced-bf01-b280472b53f1-kube-api-access-pxb9x\") pod \"migrator-66df5d496b-v6qdv\" (UID: \"356632db-499f-4ced-bf01-b280472b53f1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v6qdv"
Nov 03 21:14:42.625226 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.625171    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-g82mg-mqwqf" podStartSLOduration=1.625156923 podStartE2EDuration="1.625156923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:42.623553967 +0000 UTC m=+38.922180747" watchObservedRunningTime="2026-11-03 21:14:42.625156923 +0000 UTC m=+38.923783703"
Nov 03 21:14:42.648852 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.648528    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jchm\" (UniqueName: \"kubernetes.io/projected/2000161a-2aa8-4638-a4f7-fbe305e929ae-kube-api-access-9jchm\") pod \"migrator-66df5d496b-jqkbh\" (UID: \"2000161a-2aa8-4638-a4f7-fbe305e929ae\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jqkbh"
Nov 03 21:14:42.648852 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.648692    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxb9x\" (UniqueName: \"kubernetes.io/projected/356632db-499f-4ced-bf01-b280472b53f1-kube-api-access-pxb9x\") pod \"migrator-66df5d496b-v6qdv\" (UID: \"356632db-499f-4ced-bf01-b280472b53f1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v6qdv"
Nov 03 21:14:42.705972 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.705932    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jchm\" (UniqueName: \"kubernetes.io/projected/2000161a-2aa8-4638-a4f7-fbe305e929ae-kube-api-access-9jchm\") pod \"migrator-66df5d496b-jqkbh\" (UID: \"2000161a-2aa8-4638-a4f7-fbe305e929ae\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jqkbh"
Nov 03 21:14:42.706114 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.706009    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxb9x\" (UniqueName: \"kubernetes.io/projected/356632db-499f-4ced-bf01-b280472b53f1-kube-api-access-pxb9x\") pod \"migrator-66df5d496b-v6qdv\" (UID: \"356632db-499f-4ced-bf01-b280472b53f1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v6qdv"
Nov 03 21:14:42.746316 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.746280    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v6qdv"
Nov 03 21:14:42.764592 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.764562    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jqkbh"
Nov 03 21:14:42.862941 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.862503    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8w5st" podStartSLOduration=34.455902742 podStartE2EDuration="38.862488458s" totalImagesPullingTime="4.406585716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:42.861167069 +0000 UTC m=+39.159793850" watchObservedRunningTime="2026-11-03 21:14:42.862488458 +0000 UTC m=+39.161115238"
Nov 03 21:14:42.978141 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:42.978072    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-m5s9b" podStartSLOduration=2.566189127 podStartE2EDuration="8.978057083s" totalImagesPullingTime="6.411867956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:42.976180962 +0000 UTC m=+39.274807743" watchObservedRunningTime="2026-11-03 21:14:42.978057083 +0000 UTC m=+39.276683860"
Nov 03 21:14:43.141672 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.141637    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-v6qdv"]
Nov 03 21:14:43.145164 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:43.145126    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356632db_499f_4ced_bf01_b280472b53f1.slice/crio-66bc4eaa57d92441818ddc2ee8a2a59f5f8676389ea6efbe33c1732ea677be6f WatchSource:0}: Error finding container 66bc4eaa57d92441818ddc2ee8a2a59f5f8676389ea6efbe33c1732ea677be6f: Status 404 returned error can't find the container with id 66bc4eaa57d92441818ddc2ee8a2a59f5f8676389ea6efbe33c1732ea677be6f
Nov 03 21:14:43.157972 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.157942    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:43.158080 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:43.158056    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 21:14:43.158136 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:43.158126    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls podName:cad3df5d-3fe9-449b-8b70-793bceb47985 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:45.158107451 +0000 UTC m=+41.456734232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9vxl" (UID: "cad3df5d-3fe9-449b-8b70-793bceb47985") : secret "insights-runtime-extractor-tls" not found
Nov 03 21:14:43.158203 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.158152    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cad3df5d-3fe9-449b-8b70-793bceb47985-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:43.158336 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:43.158318    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 21:14:43.158394 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:43.158380    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cad3df5d-3fe9-449b-8b70-793bceb47985-kube-rbac-proxy-cm podName:cad3df5d-3fe9-449b-8b70-793bceb47985 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:45.158363979 +0000 UTC m=+41.456990740 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cad3df5d-3fe9-449b-8b70-793bceb47985-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v9vxl" (UID: "cad3df5d-3fe9-449b-8b70-793bceb47985") : configmap "kube-rbac-proxy" not found
Nov 03 21:14:43.184371 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:43.184335    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2000161a_2aa8_4638_a4f7_fbe305e929ae.slice/crio-a049d6608982bef0d437cb3ab400209dede225f9402b482ecb8347b30217549e WatchSource:0}: Error finding container a049d6608982bef0d437cb3ab400209dede225f9402b482ecb8347b30217549e: Status 404 returned error can't find the container with id a049d6608982bef0d437cb3ab400209dede225f9402b482ecb8347b30217549e
Nov 03 21:14:43.186721 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.186691    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jqkbh"]
Nov 03 21:14:43.493804 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.493773    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-zf7tn"]
Nov 03 21:14:43.525646 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.525605    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sb5xl" event={"ID":"06a672d7-0667-4bab-98ee-91dd71dfc489","Type":"ContainerStarted","Data":"b89b876d9988a3963cc72c3efc92de30860efd3a492654f0cee902231627c06a"}
Nov 03 21:14:43.525822 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.525656    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jqkbh" event={"ID":"2000161a-2aa8-4638-a4f7-fbe305e929ae","Type":"ContainerStarted","Data":"a049d6608982bef0d437cb3ab400209dede225f9402b482ecb8347b30217549e"}
Nov 03 21:14:43.525822 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.525746    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-zf7tn"
Nov 03 21:14:43.525822 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.525773    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v6qdv" event={"ID":"356632db-499f-4ced-bf01-b280472b53f1","Type":"ContainerStarted","Data":"66bc4eaa57d92441818ddc2ee8a2a59f5f8676389ea6efbe33c1732ea677be6f"}
Nov 03 21:14:43.552712 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.552680    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-zf7tn"]
Nov 03 21:14:43.580101 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.580027    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 21:14:43.580101 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.580028    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-smlpl\""
Nov 03 21:14:43.580101 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.580068    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 21:14:43.580365 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.580123    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 21:14:43.580365 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.580223    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 21:14:43.625822 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.624313    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sb5xl" podStartSLOduration=406.180482594 podStartE2EDuration="6m53.624300433s" totalImagesPullingTime="7.443817839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:43.624204587 +0000 UTC m=+39.922831370" watchObservedRunningTime="2026-11-03 21:14:43.624300433 +0000 UTC m=+39.922927258"
Nov 03 21:14:43.663845 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.663750    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/440dc059-df07-4f3e-a595-50d35d172e8c-signing-key\") pod \"service-ca-5fffc97bdf-zf7tn\" (UID: \"440dc059-df07-4f3e-a595-50d35d172e8c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zf7tn"
Nov 03 21:14:43.664025 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.663935    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/440dc059-df07-4f3e-a595-50d35d172e8c-signing-cabundle\") pod \"service-ca-5fffc97bdf-zf7tn\" (UID: \"440dc059-df07-4f3e-a595-50d35d172e8c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zf7tn"
Nov 03 21:14:43.664087 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.664064    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4hxv\" (UniqueName: \"kubernetes.io/projected/440dc059-df07-4f3e-a595-50d35d172e8c-kube-api-access-f4hxv\") pod \"service-ca-5fffc97bdf-zf7tn\" (UID: \"440dc059-df07-4f3e-a595-50d35d172e8c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zf7tn"
Nov 03 21:14:43.766721 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.765598    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4hxv\" (UniqueName: \"kubernetes.io/projected/440dc059-df07-4f3e-a595-50d35d172e8c-kube-api-access-f4hxv\") pod \"service-ca-5fffc97bdf-zf7tn\" (UID: \"440dc059-df07-4f3e-a595-50d35d172e8c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zf7tn"
Nov 03 21:14:43.766721 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.765821    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/440dc059-df07-4f3e-a595-50d35d172e8c-signing-key\") pod \"service-ca-5fffc97bdf-zf7tn\" (UID: \"440dc059-df07-4f3e-a595-50d35d172e8c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zf7tn"
Nov 03 21:14:43.766721 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.765850    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/440dc059-df07-4f3e-a595-50d35d172e8c-signing-cabundle\") pod \"service-ca-5fffc97bdf-zf7tn\" (UID: \"440dc059-df07-4f3e-a595-50d35d172e8c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zf7tn"
Nov 03 21:14:43.766721 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.766665    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/440dc059-df07-4f3e-a595-50d35d172e8c-signing-cabundle\") pod \"service-ca-5fffc97bdf-zf7tn\" (UID: \"440dc059-df07-4f3e-a595-50d35d172e8c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zf7tn"
Nov 03 21:14:43.768974 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.768921    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/440dc059-df07-4f3e-a595-50d35d172e8c-signing-key\") pod \"service-ca-5fffc97bdf-zf7tn\" (UID: \"440dc059-df07-4f3e-a595-50d35d172e8c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zf7tn"
Nov 03 21:14:43.823502 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.823470    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4hxv\" (UniqueName: \"kubernetes.io/projected/440dc059-df07-4f3e-a595-50d35d172e8c-kube-api-access-f4hxv\") pod \"service-ca-5fffc97bdf-zf7tn\" (UID: \"440dc059-df07-4f3e-a595-50d35d172e8c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zf7tn"
Nov 03 21:14:43.850373 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:43.850294    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-zf7tn"
Nov 03 21:14:44.245134 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:44.245060    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-zf7tn"]
Nov 03 21:14:44.250018 ip-10-0-128-43 kubenswrapper[2574]: W1103 21:14:44.249986    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440dc059_df07_4f3e_a595_50d35d172e8c.slice/crio-7522c55f6438d270d73469d5fec09f76de9bbb452ef58166233fbb310c00725f WatchSource:0}: Error finding container 7522c55f6438d270d73469d5fec09f76de9bbb452ef58166233fbb310c00725f: Status 404 returned error can't find the container with id 7522c55f6438d270d73469d5fec09f76de9bbb452ef58166233fbb310c00725f
Nov 03 21:14:44.520049 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:44.519944    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-zf7tn" event={"ID":"440dc059-df07-4f3e-a595-50d35d172e8c","Type":"ContainerStarted","Data":"c52bc4a50dbde97fdc90f2df6fe55413e36e615c630dfd87496c337f743b9ffd"}
Nov 03 21:14:44.520481 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:44.520064    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-zf7tn" event={"ID":"440dc059-df07-4f3e-a595-50d35d172e8c","Type":"ContainerStarted","Data":"7522c55f6438d270d73469d5fec09f76de9bbb452ef58166233fbb310c00725f"}
Nov 03 21:14:44.529279 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:44.529227    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:44.529428 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:44.529306    2574 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:44.529695 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:44.529619    2574 scope.go:122] "RemoveContainer" containerID="9f6b6dc724c5973a7c58dc058c00d7b81472be8fbb45b2fedb085ed90d0ec098"
Nov 03 21:14:44.529830 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:44.529807    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rhpjv_openshift-console-operator(6547c71f-71a2-40c9-912f-9283b6bdb560)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv" podUID="6547c71f-71a2-40c9-912f-9283b6bdb560"
Nov 03 21:14:44.637026 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:44.636967    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-zf7tn" podStartSLOduration=1.63695015 podStartE2EDuration="1.63695015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:44.635730738 +0000 UTC m=+40.934357519" watchObservedRunningTime="2026-11-03 21:14:44.63695015 +0000 UTC m=+40.935576929"
Nov 03 21:14:45.183978 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:45.183950    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cad3df5d-3fe9-449b-8b70-793bceb47985-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:45.184099 ip-10-0-128-43 kubenswrapper[2574]: I1103 21:14:45.184053    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:45.184099 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:45.184066    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 21:14:45.184182 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:45.184133    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cad3df5d-3fe9-449b-8b70-793bceb47985-kube-rbac-proxy-cm podName:cad3df5d-3fe9-449b-8b70-793bceb47985 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:49.184114361 +0000 UTC m=+45.482741142 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cad3df5d-3fe9-449b-8b70-793bceb47985-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v9vxl" (UID: "cad3df5d-3fe9-449b-8b70-793bceb47985") : configmap "kube-rbac-proxy" not found
Nov 03 21:14:45.184220 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:45.184188    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 21:14:45.184275 ip-10-0-128-43 kubenswrapper[2574]: E1103 21:14:45.184225    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls podName:cad3df5d-3fe9-449b-8b70-793bceb47985 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:49.184217303 +0000 UTC m=+45.482844081 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9vxl" (UID: "cad3df5d-3fe9-449b-8b70-793bceb47985") : secret "insights-runtime-extractor-tls" not found
Nov 03 21:14:45.424101 ip-10-0-128-43 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 21:14:45.439637 ip-10-0-128-43 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 21:14:45.439813 ip-10-0-128-43 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 21:14:45.439868 ip-10-0-128-43 systemd[1]: kubelet.service: Consumed 3.330s CPU time.
Nov 03 21:14:45.450853 ip-10-0-128-43 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 21:14:45.543530 ip-10-0-128-43 kubenswrapper[5053]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 21:14:45.543530 ip-10-0-128-43 kubenswrapper[5053]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 21:14:45.543530 ip-10-0-128-43 kubenswrapper[5053]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 21:14:45.543530 ip-10-0-128-43 kubenswrapper[5053]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 21:14:45.545616 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545528    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:14:45.545616 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545616    5053 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:14:45.545692 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545621    5053 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:14:45.545692 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545624    5053 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:14:45.545692 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545628    5053 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:14:45.545692 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545630    5053 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:14:45.545692 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545633    5053 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:14:45.545692 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545636    5053 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:14:45.545692 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545640    5053 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:14:45.545692 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545645    5053 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:14:45.545692 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545648    5053 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:14:45.545692 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545652    5053 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:14:45.545692 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545655    5053 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:14:45.545692 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545658    5053 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:14:45.545692 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545660    5053 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:14:45.545692 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545663    5053 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:14:45.545692 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545665    5053 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:14:45.545692 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545668    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:14:45.545692 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545670    5053 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:14:45.545692 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545673    5053 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:14:45.545692 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545675    5053 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:14:45.546157 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545678    5053 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:14:45.546157 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545680    5053 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:14:45.546157 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545683    5053 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:14:45.546157 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545685    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:14:45.546157 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545688    5053 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:14:45.546157 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545690    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:14:45.546157 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545702    5053 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:14:45.546157 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545705    5053 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:14:45.546157 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545708    5053 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:14:45.546157 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545710    5053 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:14:45.546157 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545713    5053 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:14:45.546157 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545715    5053 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:14:45.546157 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545718    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:14:45.546157 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545720    5053 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:14:45.546157 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545722    5053 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:14:45.546157 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545725    5053 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:14:45.546157 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545727    5053 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:14:45.546157 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545730    5053 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:14:45.546157 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545732    5053 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:14:45.546157 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545735    5053 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:14:45.546668 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545737    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:14:45.546668 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545739    5053 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:14:45.546668 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545742    5053 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:14:45.546668 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545744    5053 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:14:45.546668 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545747    5053 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:14:45.546668 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545749    5053 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:14:45.546668 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545752    5053 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:14:45.546668 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545755    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:14:45.546668 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545758    5053 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:14:45.546668 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545760    5053 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:14:45.546668 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545763    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:14:45.546668 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545766    5053 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:14:45.546668 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545769    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:14:45.546668 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545771    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:14:45.546668 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545774    5053 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:14:45.546668 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545777    5053 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:14:45.546668 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545779    5053 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:14:45.546668 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545781    5053 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:14:45.546668 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545785    5053 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:14:45.547126 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545788    5053 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:14:45.547126 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545790    5053 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:14:45.547126 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545793    5053 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:14:45.547126 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545795    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:14:45.547126 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545798    5053 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:14:45.547126 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545800    5053 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:14:45.547126 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545804    5053 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:14:45.547126 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545808    5053 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:14:45.547126 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545810    5053 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:14:45.547126 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545813    5053 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:14:45.547126 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545815    5053 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:14:45.547126 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545818    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:14:45.547126 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545820    5053 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:14:45.547126 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545823    5053 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:14:45.547126 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545825    5053 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:14:45.547126 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545829    5053 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:14:45.547126 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545832    5053 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:14:45.547126 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545834    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:14:45.547126 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545836    5053 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:14:45.547606 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545839    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:14:45.547606 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545842    5053 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:14:45.547606 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545844    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:14:45.547606 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545847    5053 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:14:45.547606 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545850    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:14:45.547606 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545852    5053 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:14:45.547606 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545855    5053 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:14:45.547606 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545857    5053 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:14:45.547606 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545860    5053 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:14:45.547606 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545862    5053 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:14:45.547606 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545865    5053 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:14:45.547606 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545867    5053 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:14:45.547606 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545870    5053 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:14:45.547606 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545873    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:14:45.547606 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545876    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:14:45.547606 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545878    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:14:45.547606 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545881    5053 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:14:45.547606 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545883    5053 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:14:45.547606 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545886    5053 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:14:45.547606 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545889    5053 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:14:45.548083 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545893    5053 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:14:45.548083 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545895    5053 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:14:45.548083 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545898    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:14:45.548083 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545900    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:14:45.548083 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545902    5053 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:14:45.548083 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545905    5053 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:14:45.548083 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.545907    5053 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:14:45.548083 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546695    5053 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:14:45.548083 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546703    5053 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:14:45.548083 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546705    5053 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:14:45.548083 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546709    5053 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:14:45.548083 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546711    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:14:45.548083 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546715    5053 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:14:45.548083 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546720    5053 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:14:45.548083 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546724    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:14:45.548083 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546728    5053 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:14:45.548083 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546731    5053 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:14:45.548083 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546734    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:14:45.548570 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546737    5053 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:14:45.548570 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546739    5053 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:14:45.548570 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546742    5053 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:14:45.548570 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546745    5053 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:14:45.548570 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546747    5053 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:14:45.548570 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546750    5053 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:14:45.548570 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546752    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:14:45.548570 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546755    5053 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:14:45.548570 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546758    5053 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:14:45.548570 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546761    5053 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:14:45.548570 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546763    5053 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:14:45.548570 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546766    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:14:45.548570 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546769    5053 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:14:45.548570 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546771    5053 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:14:45.548570 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546774    5053 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:14:45.548570 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546776    5053 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:14:45.548570 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546779    5053 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:14:45.548570 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546781    5053 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:14:45.548570 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546784    5053 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:14:45.548570 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546786    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:14:45.549037 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546788    5053 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:14:45.549037 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546791    5053 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:14:45.549037 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546794    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:14:45.549037 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546797    5053 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:14:45.549037 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546800    5053 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:14:45.549037 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546803    5053 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:14:45.549037 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546805    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:14:45.549037 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546808    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:14:45.549037 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546811    5053 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:14:45.549037 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546813    5053 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:14:45.549037 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546815    5053 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:14:45.549037 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546818    5053 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:14:45.549037 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546821    5053 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:14:45.549037 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546823    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:14:45.549037 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546830    5053 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:14:45.549037 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546832    5053 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:14:45.549037 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546835    5053 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:14:45.549037 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546838    5053 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:14:45.549037 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546841    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:14:45.549555 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546843    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:14:45.549555 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546846    5053 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:14:45.549555 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546849    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:14:45.549555 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546851    5053 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:14:45.549555 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546854    5053 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:14:45.549555 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546856    5053 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:14:45.549555 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546859    5053 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:14:45.549555 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546862    5053 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:14:45.549555 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546865    5053 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:14:45.549555 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546868    5053 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:14:45.549555 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546871    5053 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:14:45.549555 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546873    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:14:45.549555 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546876    5053 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:14:45.549555 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546878    5053 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:14:45.549555 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546881    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:14:45.549555 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546884    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:14:45.549555 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546887    5053 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:14:45.549555 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546889    5053 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:14:45.549555 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546892    5053 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:14:45.549555 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546894    5053 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:14:45.550058 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546897    5053 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:14:45.550058 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546899    5053 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:14:45.550058 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546902    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:14:45.550058 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546904    5053 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:14:45.550058 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546907    5053 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:14:45.550058 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546909    5053 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:14:45.550058 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546912    5053 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:14:45.550058 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546914    5053 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:14:45.550058 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546917    5053 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:14:45.550058 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546919    5053 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:14:45.550058 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546922    5053 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:14:45.550058 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546925    5053 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:14:45.550058 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546927    5053 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:14:45.550058 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546929    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:14:45.550058 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546932    5053 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:14:45.550058 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546935    5053 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:14:45.550058 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546938    5053 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:14:45.550058 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546941    5053 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:14:45.550058 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546943    5053 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:14:45.550548 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546946    5053 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:14:45.550548 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546949    5053 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:14:45.550548 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546953    5053 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:14:45.550548 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546956    5053 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:14:45.550548 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546959    5053 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:14:45.550548 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546961    5053 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:14:45.550548 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546964    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:14:45.550548 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546966    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:14:45.550548 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546968    5053 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:14:45.550548 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546971    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:14:45.550548 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546973    5053 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:14:45.550548 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546976    5053 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:14:45.550548 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546978    5053 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:14:45.550548 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546981    5053 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:14:45.550548 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546983    5053 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:14:45.550548 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546986    5053 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:14:45.550548 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.546989    5053 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:14:45.550548 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547502    5053 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:14:45.550548 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547509    5053 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:14:45.551046 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547512    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:14:45.551046 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547515    5053 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:14:45.551046 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547518    5053 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:14:45.551046 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547520    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:14:45.551046 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547523    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:14:45.551046 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547526    5053 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:14:45.551046 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547529    5053 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:14:45.551046 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547531    5053 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:14:45.551046 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547534    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:14:45.551046 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547537    5053 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:14:45.551046 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547539    5053 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:14:45.551046 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547542    5053 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:14:45.551046 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547545    5053 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:14:45.551046 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547547    5053 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:14:45.551046 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547550    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:14:45.551046 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547552    5053 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:14:45.551046 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547555    5053 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:14:45.551046 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547557    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:14:45.551046 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547560    5053 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:14:45.551046 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547562    5053 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:14:45.551549 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547565    5053 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:14:45.551549 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547567    5053 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:14:45.551549 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547570    5053 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:14:45.551549 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547575    5053 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:14:45.551549 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547578    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:14:45.551549 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547580    5053 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:14:45.551549 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547583    5053 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:14:45.551549 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547585    5053 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:14:45.551549 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547588    5053 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:14:45.551549 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547590    5053 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:14:45.551549 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547593    5053 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:14:45.551549 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547595    5053 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:14:45.551549 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547598    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:14:45.551549 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547600    5053 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:14:45.551549 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547603    5053 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:14:45.551549 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547605    5053 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:14:45.551549 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547608    5053 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:14:45.551549 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547610    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:14:45.551549 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547613    5053 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:14:45.552008 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547616    5053 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:14:45.552008 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547618    5053 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:14:45.552008 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547621    5053 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:14:45.552008 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547624    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:14:45.552008 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547627    5053 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:14:45.552008 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547630    5053 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:14:45.552008 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547632    5053 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:14:45.552008 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547635    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:14:45.552008 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547637    5053 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:14:45.552008 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547640    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:14:45.552008 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547642    5053 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:14:45.552008 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547646    5053 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:14:45.552008 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547649    5053 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:14:45.552008 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547652    5053 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:14:45.552008 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547654    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:14:45.552008 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547657    5053 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:14:45.552008 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547661    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:14:45.552008 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547664    5053 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:14:45.552008 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547666    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:14:45.552480 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547668    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:14:45.552480 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547671    5053 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:14:45.552480 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547674    5053 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:14:45.552480 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547676    5053 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:14:45.552480 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547679    5053 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:14:45.552480 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547681    5053 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:14:45.552480 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547684    5053 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:14:45.552480 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547686    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:14:45.552480 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547688    5053 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:14:45.552480 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547691    5053 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:14:45.552480 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547693    5053 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:14:45.552480 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547696    5053 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:14:45.552480 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547698    5053 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:14:45.552480 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547701    5053 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:14:45.552480 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547704    5053 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:14:45.552480 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547706    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:14:45.552480 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547708    5053 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:14:45.552480 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547712    5053 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:14:45.552480 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547714    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:14:45.552919 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547718    5053 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:14:45.552919 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547721    5053 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:14:45.552919 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547724    5053 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:14:45.552919 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547726    5053 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:14:45.552919 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547729    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:14:45.552919 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547731    5053 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:14:45.552919 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547734    5053 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:14:45.552919 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547736    5053 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:14:45.552919 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547739    5053 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:14:45.552919 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547742    5053 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:14:45.552919 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547746    5053 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:14:45.552919 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547748    5053 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:14:45.552919 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547751    5053 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:14:45.552919 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547753    5053 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:14:45.552919 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547756    5053 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:14:45.552919 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547759    5053 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:14:45.552919 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547762    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:14:45.552919 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547764    5053 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:14:45.552919 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547767    5053 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:14:45.553390 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547769    5053 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:14:45.553390 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547771    5053 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:14:45.553390 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547774    5053 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:14:45.553390 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547776    5053 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:14:45.553390 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547779    5053 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:14:45.553390 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547781    5053 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:14:45.553390 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547784    5053 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:14:45.553390 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.547786    5053 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:14:45.553390 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547877    5053 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 21:14:45.553390 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547884    5053 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 21:14:45.553390 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547891    5053 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 21:14:45.553390 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547896    5053 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 21:14:45.553390 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547901    5053 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 21:14:45.553390 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547904    5053 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 21:14:45.553390 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547908    5053 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 21:14:45.553390 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547913    5053 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 21:14:45.553390 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547916    5053 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 21:14:45.553390 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547919    5053 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 21:14:45.553390 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547923    5053 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 21:14:45.553390 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547926    5053 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 21:14:45.553390 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547929    5053 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 21:14:45.553390 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547932    5053 flags.go:64] FLAG: --cgroup-root=""
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547936    5053 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547941    5053 flags.go:64] FLAG: --client-ca-file=""
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547944    5053 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547947    5053 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547951    5053 flags.go:64] FLAG: --cluster-domain=""
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547954    5053 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547957    5053 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547960    5053 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547964    5053 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547968    5053 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547971    5053 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547974    5053 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547977    5053 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547980    5053 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547983    5053 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547986    5053 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547989    5053 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547992    5053 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547996    5053 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.547999    5053 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548002    5053 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548005    5053 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548008    5053 flags.go:64] FLAG: --enable-server="true"
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548011    5053 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 21:14:45.553919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548016    5053 flags.go:64] FLAG: --event-burst="100"
Nov 03 21:14:45.554541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548020    5053 flags.go:64] FLAG: --event-qps="50"
Nov 03 21:14:45.554541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548023    5053 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 21:14:45.554541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548026    5053 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 21:14:45.554541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548029    5053 flags.go:64] FLAG: --eviction-hard=""
Nov 03 21:14:45.554541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548034    5053 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 21:14:45.554541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548037    5053 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 21:14:45.554541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548040    5053 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 21:14:45.554541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548044    5053 flags.go:64] FLAG: --eviction-soft=""
Nov 03 21:14:45.554541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548049    5053 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 21:14:45.554541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548052    5053 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 21:14:45.554541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548055    5053 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 21:14:45.554541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548058    5053 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 21:14:45.554541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548061    5053 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 21:14:45.554541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548063    5053 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 21:14:45.554541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548066    5053 flags.go:64] FLAG: --feature-gates=""
Nov 03 21:14:45.554541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548070    5053 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 21:14:45.554541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548074    5053 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 21:14:45.554541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548077    5053 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 21:14:45.554541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548080    5053 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 21:14:45.554541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548083    5053 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 21:14:45.554541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548086    5053 flags.go:64] FLAG: --help="false"
Nov 03 21:14:45.554541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548090    5053 flags.go:64] FLAG: --hostname-override="ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:45.554541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548093    5053 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 21:14:45.554541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548096    5053 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 21:14:45.555115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548099    5053 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 21:14:45.555115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548102    5053 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 21:14:45.555115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548106    5053 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 21:14:45.555115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548109    5053 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 21:14:45.555115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548111    5053 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 21:14:45.555115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548114    5053 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 21:14:45.555115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548117    5053 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 21:14:45.555115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548120    5053 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 21:14:45.555115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548123    5053 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 21:14:45.555115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548126    5053 flags.go:64] FLAG: --kube-reserved=""
Nov 03 21:14:45.555115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548129    5053 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 21:14:45.555115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548132    5053 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 21:14:45.555115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548135    5053 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 21:14:45.555115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548138    5053 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 21:14:45.555115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548141    5053 flags.go:64] FLAG: --lock-file=""
Nov 03 21:14:45.555115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548145    5053 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 21:14:45.555115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548149    5053 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 21:14:45.555115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548152    5053 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 21:14:45.555115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548158    5053 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 21:14:45.555115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548161    5053 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 21:14:45.555115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548164    5053 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 21:14:45.555115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548166    5053 flags.go:64] FLAG: --logging-format="text"
Nov 03 21:14:45.555115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548169    5053 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548173    5053 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548176    5053 flags.go:64] FLAG: --manifest-url=""
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548179    5053 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548183    5053 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548186    5053 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548190    5053 flags.go:64] FLAG: --max-pods="110"
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548193    5053 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548196    5053 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548199    5053 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548203    5053 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548206    5053 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548209    5053 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548212    5053 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548219    5053 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548222    5053 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548225    5053 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548228    5053 flags.go:64] FLAG: --pod-cidr=""
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548247    5053 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548252    5053 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548256    5053 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548261    5053 flags.go:64] FLAG: --port="10250"
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548265    5053 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548269    5053 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-044a84e5a53399cbf"
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548275    5053 flags.go:64] FLAG: --qos-reserved=""
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548280    5053 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 21:14:45.555703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548284    5053 flags.go:64] FLAG: --register-node="true"
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548287    5053 flags.go:64] FLAG: --register-with-taints=""
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548291    5053 flags.go:64] FLAG: --registry-burst="10"
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548294    5053 flags.go:64] FLAG: --registry-qps="5"
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548297    5053 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548300    5053 flags.go:64] FLAG: --reserved-memory=""
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548304    5053 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548307    5053 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548310    5053 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548313    5053 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548316    5053 flags.go:64] FLAG: --runonce="false"
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548319    5053 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548322    5053 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548325    5053 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548328    5053 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548331    5053 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548334    5053 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548337    5053 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548340    5053 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548343    5053 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548346    5053 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548349    5053 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548352    5053 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548354    5053 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548357    5053 flags.go:64] FLAG: --system-cgroups=""
Nov 03 21:14:45.556342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548361    5053 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 21:14:45.557332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548366    5053 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 21:14:45.557332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548369    5053 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 21:14:45.557332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548372    5053 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 21:14:45.557332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548403    5053 flags.go:64] FLAG: --tls-min-version=""
Nov 03 21:14:45.557332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548406    5053 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 21:14:45.557332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548411    5053 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 21:14:45.557332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548416    5053 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 21:14:45.557332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548419    5053 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 21:14:45.557332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548422    5053 flags.go:64] FLAG: --v="2"
Nov 03 21:14:45.557332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548426    5053 flags.go:64] FLAG: --version="false"
Nov 03 21:14:45.557332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548431    5053 flags.go:64] FLAG: --vmodule=""
Nov 03 21:14:45.557332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548435    5053 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 21:14:45.557332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548438    5053 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 21:14:45.557332 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548594    5053 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:14:45.557332 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548599    5053 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:14:45.557332 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548602    5053 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:14:45.557332 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548604    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:14:45.557332 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548607    5053 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:14:45.557332 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548609    5053 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:14:45.557332 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548612    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:14:45.557332 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548614    5053 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:14:45.557332 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548617    5053 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:14:45.557332 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548619    5053 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:14:45.557969 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548622    5053 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:14:45.557969 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548624    5053 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:14:45.557969 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548627    5053 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:14:45.557969 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548629    5053 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:14:45.557969 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548632    5053 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:14:45.557969 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548635    5053 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:14:45.557969 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548637    5053 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:14:45.557969 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548639    5053 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:14:45.557969 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548644    5053 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:14:45.557969 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548648    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:14:45.557969 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548651    5053 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:14:45.557969 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548654    5053 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:14:45.557969 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548657    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:14:45.557969 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548659    5053 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:14:45.557969 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548664    5053 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:14:45.557969 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548669    5053 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:14:45.557969 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548671    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:14:45.557969 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548674    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:14:45.557969 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548676    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:14:45.558458 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548679    5053 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:14:45.558458 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548682    5053 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:14:45.558458 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548684    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:14:45.558458 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548686    5053 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:14:45.558458 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548689    5053 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:14:45.558458 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548691    5053 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:14:45.558458 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548694    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:14:45.558458 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548696    5053 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:14:45.558458 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548699    5053 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:14:45.558458 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548702    5053 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:14:45.558458 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548704    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:14:45.558458 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548706    5053 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:14:45.558458 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548709    5053 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:14:45.558458 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548711    5053 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:14:45.558458 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548714    5053 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:14:45.558458 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548716    5053 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:14:45.558458 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548719    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:14:45.558458 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548721    5053 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:14:45.558458 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548723    5053 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:14:45.558458 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548726    5053 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:14:45.558943 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548728    5053 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:14:45.558943 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548730    5053 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:14:45.558943 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548733    5053 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:14:45.558943 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548736    5053 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:14:45.558943 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548739    5053 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:14:45.558943 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548741    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:14:45.558943 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548744    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:14:45.558943 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548748    5053 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:14:45.558943 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548754    5053 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:14:45.558943 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548756    5053 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:14:45.558943 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548759    5053 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:14:45.558943 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548762    5053 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:14:45.558943 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548764    5053 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:14:45.558943 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548767    5053 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:14:45.558943 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548769    5053 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:14:45.558943 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548772    5053 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:14:45.558943 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548774    5053 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:14:45.558943 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548777    5053 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:14:45.558943 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548780    5053 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:14:45.558943 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548782    5053 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:14:45.559595 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548785    5053 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:14:45.559595 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548787    5053 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:14:45.559595 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548790    5053 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:14:45.559595 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548792    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:14:45.559595 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548795    5053 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:14:45.559595 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548797    5053 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:14:45.559595 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548800    5053 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:14:45.559595 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548802    5053 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:14:45.559595 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548805    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:14:45.559595 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548807    5053 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:14:45.559595 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548810    5053 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:14:45.559595 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548813    5053 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:14:45.559595 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548815    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:14:45.559595 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548818    5053 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:14:45.559595 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548820    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:14:45.559595 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548823    5053 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:14:45.559595 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548826    5053 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:14:45.559595 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548828    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:14:45.559595 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548831    5053 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:14:45.560163 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548835    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:14:45.560163 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548839    5053 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:14:45.560163 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548842    5053 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:14:45.560163 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548844    5053 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:14:45.560163 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548847    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:14:45.560163 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548849    5053 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:14:45.560163 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548852    5053 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:14:45.560163 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548854    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:14:45.560163 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548857    5053 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:14:45.560163 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548859    5053 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:14:45.560163 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548862    5053 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:14:45.560163 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548864    5053 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:14:45.560163 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548867    5053 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:14:45.560163 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548869    5053 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:14:45.560163 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548873    5053 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:14:45.560163 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548877    5053 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:14:45.560163 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548880    5053 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:14:45.560163 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548883    5053 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:14:45.560765 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.548909    5053 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 21:14:45.560765 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548946    5053 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:14:45.560765 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548951    5053 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:14:45.560765 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548954    5053 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:14:45.560765 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548956    5053 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:14:45.560765 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548960    5053 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:14:45.560765 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548963    5053 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:14:45.560765 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548965    5053 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:14:45.560765 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548968    5053 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:14:45.560765 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548972    5053 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:14:45.560765 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548977    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:14:45.560765 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548980    5053 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:14:45.560765 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548983    5053 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:14:45.560765 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548986    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:14:45.560765 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548990    5053 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:14:45.560765 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548993    5053 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:14:45.561202 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.548997    5053 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:14:45.561202 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549000    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:14:45.561202 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549003    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:14:45.561202 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549005    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:14:45.561202 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549008    5053 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:14:45.561202 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549010    5053 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:14:45.561202 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549013    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:14:45.561202 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549015    5053 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:14:45.561202 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549017    5053 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:14:45.561202 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549020    5053 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:14:45.561202 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549023    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:14:45.561202 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549025    5053 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:14:45.561202 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549028    5053 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:14:45.561202 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549030    5053 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:14:45.561202 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549033    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:14:45.561202 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549035    5053 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:14:45.561202 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549038    5053 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:14:45.561202 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549040    5053 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:14:45.561202 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549043    5053 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:14:45.561202 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549046    5053 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:14:45.561745 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549048    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:14:45.561745 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549051    5053 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:14:45.561745 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549054    5053 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:14:45.561745 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549056    5053 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:14:45.561745 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549059    5053 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:14:45.561745 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549061    5053 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:14:45.561745 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549064    5053 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:14:45.561745 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549066    5053 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:14:45.561745 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549069    5053 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:14:45.561745 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549072    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:14:45.561745 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549076    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:14:45.561745 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549078    5053 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:14:45.561745 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549081    5053 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:14:45.561745 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549084    5053 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:14:45.561745 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549086    5053 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:14:45.561745 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549089    5053 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:14:45.561745 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549091    5053 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:14:45.561745 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549094    5053 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:14:45.561745 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549096    5053 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:14:45.562550 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549099    5053 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:14:45.562550 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549101    5053 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:14:45.562550 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549104    5053 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:14:45.562550 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549106    5053 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:14:45.562550 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549109    5053 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:14:45.562550 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549112    5053 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:14:45.562550 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549114    5053 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:14:45.562550 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549116    5053 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:14:45.562550 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549119    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:14:45.562550 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549121    5053 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:14:45.562550 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549124    5053 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:14:45.562550 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549126    5053 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:14:45.562550 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549128    5053 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:14:45.562550 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549131    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:14:45.562550 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549133    5053 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:14:45.562550 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549136    5053 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:14:45.562550 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549138    5053 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:14:45.562550 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549140    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:14:45.562550 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549143    5053 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:14:45.563080 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549145    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:14:45.563080 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549148    5053 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:14:45.563080 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549150    5053 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:14:45.563080 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549153    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:14:45.563080 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549157    5053 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:14:45.563080 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549160    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:14:45.563080 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549162    5053 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:14:45.563080 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549165    5053 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:14:45.563080 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549167    5053 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:14:45.563080 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549169    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:14:45.563080 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549172    5053 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:14:45.563080 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549174    5053 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:14:45.563080 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549177    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:14:45.563080 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549179    5053 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:14:45.563080 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549181    5053 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:14:45.563080 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549184    5053 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:14:45.563080 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549187    5053 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:14:45.563080 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549189    5053 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:14:45.563080 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549191    5053 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:14:45.563623 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549195    5053 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:14:45.563623 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549199    5053 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:14:45.563623 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549202    5053 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:14:45.563623 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549205    5053 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:14:45.563623 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549207    5053 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:14:45.563623 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549210    5053 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:14:45.563623 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549212    5053 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:14:45.563623 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549214    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:14:45.563623 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549217    5053 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:14:45.563623 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549219    5053 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:14:45.563623 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549222    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:14:45.563623 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549224    5053 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:14:45.563623 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549227    5053 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:14:45.563623 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.549229    5053 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:14:45.563623 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.553166    5053 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 21:14:45.563623 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.553182    5053 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 21:14:45.563623 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.553200    5053 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 21:14:45.563623 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.553206    5053 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 21:14:45.563623 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553288    5053 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:14:45.564076 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553295    5053 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:14:45.564076 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553298    5053 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:14:45.564076 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553302    5053 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:14:45.564076 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553304    5053 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:14:45.564076 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553307    5053 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:14:45.564076 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553310    5053 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:14:45.564076 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553312    5053 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:14:45.564076 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553315    5053 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:14:45.564076 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553317    5053 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:14:45.564076 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553320    5053 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:14:45.564076 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553323    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:14:45.564076 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553325    5053 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:14:45.564076 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553328    5053 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:14:45.564076 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553330    5053 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:14:45.564076 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553333    5053 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:14:45.564076 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553335    5053 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:14:45.564076 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553338    5053 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:14:45.564076 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553341    5053 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:14:45.564076 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553343    5053 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:14:45.564076 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553346    5053 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:14:45.565224 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553348    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:14:45.565224 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553351    5053 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:14:45.565224 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553354    5053 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:14:45.565224 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553356    5053 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:14:45.565224 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553359    5053 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:14:45.565224 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553362    5053 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:14:45.565224 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553364    5053 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:14:45.565224 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553367    5053 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:14:45.565224 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553369    5053 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:14:45.565224 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553372    5053 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:14:45.565224 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553374    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:14:45.565224 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553378    5053 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:14:45.565224 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553380    5053 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:14:45.565224 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553382    5053 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:14:45.565224 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553385    5053 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:14:45.565224 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553387    5053 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:14:45.565224 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553390    5053 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:14:45.565224 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553394    5053 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:14:45.565224 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553399    5053 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:14:45.565884 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553403    5053 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:14:45.565884 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553406    5053 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:14:45.565884 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553409    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:14:45.565884 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553412    5053 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:14:45.565884 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553415    5053 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:14:45.565884 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553418    5053 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:14:45.565884 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553420    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:14:45.565884 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553423    5053 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:14:45.565884 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553425    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:14:45.565884 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553428    5053 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:14:45.565884 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553431    5053 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:14:45.565884 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553434    5053 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:14:45.565884 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553437    5053 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:14:45.565884 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553440    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:14:45.565884 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553443    5053 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:14:45.565884 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553445    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:14:45.565884 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553448    5053 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:14:45.565884 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553451    5053 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:14:45.565884 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553453    5053 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:14:45.565884 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553456    5053 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:14:45.566374 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553459    5053 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:14:45.566374 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553461    5053 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:14:45.566374 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553464    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:14:45.566374 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553467    5053 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:14:45.566374 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553470    5053 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:14:45.566374 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553473    5053 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:14:45.566374 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553476    5053 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:14:45.566374 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553478    5053 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:14:45.566374 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553481    5053 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:14:45.566374 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553483    5053 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:14:45.566374 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553486    5053 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:14:45.566374 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553488    5053 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:14:45.566374 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553490    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:14:45.566374 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553493    5053 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:14:45.566374 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553495    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:14:45.566374 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553498    5053 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:14:45.566374 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553500    5053 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:14:45.566374 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553503    5053 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:14:45.566374 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553505    5053 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:14:45.566374 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553507    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:14:45.566865 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553510    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:14:45.566865 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553513    5053 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:14:45.566865 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553517    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:14:45.566865 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553520    5053 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:14:45.566865 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553523    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:14:45.566865 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553526    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:14:45.566865 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553528    5053 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:14:45.566865 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553531    5053 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:14:45.566865 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553533    5053 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:14:45.566865 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553536    5053 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:14:45.566865 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553539    5053 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:14:45.566865 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553541    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:14:45.566865 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553544    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:14:45.566865 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553547    5053 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:14:45.566865 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553549    5053 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:14:45.566865 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553552    5053 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:14:45.566865 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553554    5053 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:14:45.566865 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553557    5053 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:14:45.566865 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553560    5053 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:14:45.567371 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553562    5053 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:14:45.567371 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553564    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:14:45.567371 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553567    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:14:45.567371 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553569    5053 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:14:45.567371 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553572    5053 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:14:45.567371 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553574    5053 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:14:45.567371 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553577    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:14:45.567371 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.553604    5053 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 21:14:45.567371 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553753    5053 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:14:45.567371 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553758    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:14:45.567371 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553761    5053 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:14:45.567371 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553764    5053 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:14:45.567371 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553767    5053 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:14:45.567371 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553769    5053 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:14:45.567371 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553772    5053 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:14:45.567371 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553775    5053 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:14:45.567883 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553778    5053 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:14:45.567883 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553780    5053 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:14:45.567883 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553782    5053 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:14:45.567883 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553785    5053 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:14:45.567883 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553787    5053 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:14:45.567883 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553790    5053 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:14:45.567883 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553792    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:14:45.567883 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553795    5053 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:14:45.567883 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553797    5053 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:14:45.567883 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553800    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:14:45.567883 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553802    5053 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:14:45.567883 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553805    5053 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:14:45.567883 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553807    5053 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:14:45.567883 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553810    5053 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:14:45.567883 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553812    5053 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:14:45.567883 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553815    5053 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:14:45.567883 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553818    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:14:45.567883 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553821    5053 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:14:45.567883 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553823    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:14:45.568572 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553827    5053 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:14:45.568572 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553830    5053 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:14:45.568572 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553833    5053 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:14:45.568572 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553835    5053 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:14:45.568572 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553838    5053 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:14:45.568572 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553840    5053 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:14:45.568572 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553843    5053 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:14:45.568572 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553845    5053 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:14:45.568572 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553848    5053 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:14:45.568572 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553850    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:14:45.568572 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553853    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:14:45.568572 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553855    5053 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:14:45.568572 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553858    5053 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:14:45.568572 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553860    5053 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:14:45.568572 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553863    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:14:45.568572 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553865    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:14:45.568572 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553868    5053 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:14:45.568572 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553870    5053 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:14:45.568572 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553873    5053 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:14:45.568572 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553875    5053 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:14:45.569730 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553877    5053 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:14:45.569730 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553880    5053 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:14:45.569730 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553882    5053 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:14:45.569730 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553885    5053 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:14:45.569730 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553887    5053 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:14:45.569730 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553890    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:14:45.569730 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553892    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:14:45.569730 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553895    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:14:45.569730 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553897    5053 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:14:45.569730 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553900    5053 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:14:45.569730 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553906    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:14:45.569730 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553908    5053 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:14:45.569730 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553911    5053 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:14:45.569730 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553914    5053 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:14:45.569730 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553916    5053 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:14:45.569730 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553920    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:14:45.569730 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553922    5053 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:14:45.569730 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553925    5053 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:14:45.569730 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553927    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:14:45.569730 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553931    5053 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:14:45.570542 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553933    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:14:45.570542 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553936    5053 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:14:45.570542 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553938    5053 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:14:45.570542 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553941    5053 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:14:45.570542 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553943    5053 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:14:45.570542 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553946    5053 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:14:45.570542 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553948    5053 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:14:45.570542 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553951    5053 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:14:45.570542 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553953    5053 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:14:45.570542 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553956    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:14:45.570542 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553958    5053 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:14:45.570542 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553961    5053 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:14:45.570542 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553963    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:14:45.570542 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553966    5053 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:14:45.570542 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553968    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:14:45.570542 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553971    5053 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:14:45.570542 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553973    5053 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:14:45.570542 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553975    5053 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:14:45.570542 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553978    5053 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:14:45.572445 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553980    5053 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:14:45.572445 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553983    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:14:45.572445 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553985    5053 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:14:45.572445 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553991    5053 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:14:45.572445 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553995    5053 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:14:45.572445 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.553998    5053 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:14:45.572445 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554001    5053 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:14:45.572445 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554004    5053 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:14:45.572445 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554006    5053 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:14:45.572445 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554009    5053 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:14:45.572445 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554011    5053 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:14:45.572445 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554014    5053 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:14:45.572445 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554016    5053 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:14:45.572445 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554019    5053 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:14:45.572445 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554021    5053 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:14:45.572445 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554024    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:14:45.572445 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554026    5053 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:14:45.572445 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554029    5053 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:14:45.572445 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554031    5053 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:14:45.573221 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554034    5053 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:14:45.573221 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.554057    5053 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 21:14:45.573221 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554095    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:14:45.573221 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554099    5053 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:14:45.573221 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554102    5053 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:14:45.573221 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554105    5053 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:14:45.573221 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554107    5053 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:14:45.573221 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554110    5053 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:14:45.573221 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554112    5053 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:14:45.573221 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554115    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:14:45.573221 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554117    5053 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:14:45.573221 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554120    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:14:45.573221 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554124    5053 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:14:45.573221 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554127    5053 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:14:45.573221 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554130    5053 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:14:45.573879 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554132    5053 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:14:45.573879 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554135    5053 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:14:45.573879 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554138    5053 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:14:45.573879 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554141    5053 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:14:45.573879 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554144    5053 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:14:45.573879 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554147    5053 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:14:45.573879 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554149    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:14:45.573879 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554152    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:14:45.573879 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554154    5053 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:14:45.573879 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554157    5053 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:14:45.573879 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554159    5053 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:14:45.573879 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554162    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:14:45.573879 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554164    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:14:45.573879 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554167    5053 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:14:45.573879 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554169    5053 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:14:45.573879 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554172    5053 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:14:45.573879 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554174    5053 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:14:45.573879 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554177    5053 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:14:45.573879 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554179    5053 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:14:45.573879 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554182    5053 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:14:45.574682 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554184    5053 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:14:45.574682 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554187    5053 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:14:45.574682 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554189    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:14:45.574682 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554192    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:14:45.574682 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554194    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:14:45.574682 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554197    5053 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:14:45.574682 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554199    5053 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:14:45.574682 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554202    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:14:45.574682 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554204    5053 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:14:45.574682 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554206    5053 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:14:45.574682 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554209    5053 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:14:45.574682 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554211    5053 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:14:45.574682 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554214    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:14:45.574682 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554216    5053 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:14:45.574682 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554220    5053 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:14:45.574682 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554223    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:14:45.574682 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554226    5053 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:14:45.574682 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554229    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:14:45.574682 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554246    5053 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:14:45.574682 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554251    5053 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:14:45.575204 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554254    5053 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:14:45.575204 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554257    5053 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:14:45.575204 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554259    5053 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:14:45.575204 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554262    5053 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:14:45.575204 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554264    5053 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:14:45.575204 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554267    5053 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:14:45.575204 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554270    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:14:45.575204 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554272    5053 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:14:45.575204 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554275    5053 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:14:45.575204 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554277    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:14:45.575204 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554280    5053 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:14:45.575204 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554282    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:14:45.575204 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554284    5053 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:14:45.575204 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554287    5053 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:14:45.575204 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554290    5053 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:14:45.575204 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554292    5053 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:14:45.575204 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554295    5053 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:14:45.575204 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554297    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:14:45.575204 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554299    5053 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:14:45.575872 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554303    5053 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:14:45.575872 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554307    5053 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:14:45.575872 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554310    5053 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:14:45.575872 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554312    5053 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:14:45.575872 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554315    5053 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:14:45.575872 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554319    5053 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:14:45.575872 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554321    5053 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:14:45.575872 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554325    5053 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:14:45.575872 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554328    5053 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:14:45.575872 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554330    5053 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:14:45.575872 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554333    5053 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:14:45.575872 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554336    5053 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:14:45.575872 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554339    5053 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:14:45.575872 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554341    5053 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:14:45.575872 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554344    5053 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:14:45.575872 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554346    5053 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:14:45.575872 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554349    5053 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:14:45.575872 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554351    5053 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:14:45.575872 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554354    5053 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:14:45.576413 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554356    5053 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:14:45.576413 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554358    5053 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:14:45.576413 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554361    5053 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:14:45.576413 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554363    5053 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:14:45.576413 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554366    5053 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:14:45.576413 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554369    5053 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:14:45.576413 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554372    5053 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:14:45.576413 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554374    5053 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:14:45.576413 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554377    5053 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:14:45.576413 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554379    5053 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:14:45.576413 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554382    5053 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:14:45.576413 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554385    5053 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:14:45.576413 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554387    5053 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:14:45.576413 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554390    5053 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:14:45.576413 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:45.554392    5053 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:14:45.576413 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.554446    5053 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 21:14:45.576413 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.555437    5053 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 21:14:45.576413 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.555492    5053 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 21:14:45.576413 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.555679    5053 server.go:1014] "Starting client certificate rotation"
Nov 03 21:14:45.576886 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.555815    5053 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 21:14:45.576886 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.555870    5053 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 21:09:04 +0000 UTC" deadline="2028-04-21 10:56:09.140583506 +0000 UTC"
Nov 03 21:14:45.576886 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.555903    5053 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12829h41m23.584682692s"
Nov 03 21:14:45.576886 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.556147    5053 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 21:14:45.576886 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.558397    5053 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 21:14:45.576886 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.560096    5053 log.go:25] "Validated CRI v1 runtime API"
Nov 03 21:14:45.576886 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.561959    5053 log.go:25] "Validated CRI v1 image API"
Nov 03 21:14:45.576886 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.562824    5053 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 21:14:45.576886 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.564841    5053 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ba4d75bf-5393-4569-bdef-490af2af4b7e:/dev/nvme0n1p3 c0aae208-0c17-4504-90a8-de75bb41017d:/dev/nvme0n1p4]
Nov 03 21:14:45.577141 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.564866    5053 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/047235fabf11c01f841f93ff511e942d78c3561e32d9afb78b1ee150f1fcc226/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/047235fabf11c01f841f93ff511e942d78c3561e32d9afb78b1ee150f1fcc226/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/097b094bd2ef653693302fbc8d2d34b595702dd18f0ece22727dec3df5532e2e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/097b094bd2ef653693302fbc8d2d34b595702dd18f0ece22727dec3df5532e2e/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a15829a8bd8ad5d05c335ab39dbdf0c9aa529ac4f64fcb36679332864076283/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a15829a8bd8ad5d05c335ab39dbdf0c9aa529ac4f64fcb36679332864076283/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0f7c57ffd61c558ae200d75efe4615c846e43a8fbfc892cb5dec957acbfab481/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0f7c57ffd61c558ae200d75efe4615c846e43a8fbfc892cb5dec957acbfab481/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19c349dd462dcbcf83ae4a8b956193a1b9484ecd552fa9ed773e410c18d16d79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19c349dd462dcbcf83ae4a8b956193a1b9484ecd552fa9ed773e410c18d16d79/userdata/shm major:0 minor:52 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2283513bde82fc74d844a3598f31a3ab9f6a0e1b59d1aab361f8cadf549d543b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2283513bde82fc74d844a3598f31a3ab9f6a0e1b59d1aab361f8cadf549d543b/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2671cefed3cb069b2b6da7234d7d5f384a14b35dd5a2ccbe39248b441f34666b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2671cefed3cb069b2b6da7234d7d5f384a14b35dd5a2ccbe39248b441f34666b/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/324ab33cc272046022d3836ad1f36a85253684b77101adc607fdacf44b143a2b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/324ab33cc272046022d3836ad1f36a85253684b77101adc607fdacf44b143a2b/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/39fa4173aebabde72c8d3f7f8b97eae39ea8e8495f4c98be0237f5a3f3e78f0c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/39fa4173aebabde72c8d3f7f8b97eae39ea8e8495f4c98be0237f5a3f3e78f0c/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42ea59ba3e76bc7fd70fe4215d37b5fcaa705382189b846299d9f3659b476d02/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42ea59ba3e76bc7fd70fe4215d37b5fcaa705382189b846299d9f3659b476d02/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/654d663557e8f82d99dcdd5803bd5874617cadba42096308eff8d1f02824feab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/654d663557e8f82d99dcdd5803bd5874617cadba42096308eff8d1f02824feab/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/66bc4eaa57d92441818ddc2ee8a2a59f5f8676389ea6efbe33c1732ea677be6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/66bc4eaa57d92441818ddc2ee8a2a59f5f8676389ea6efbe33c1732ea677be6f/userdata/shm major:0 minor:352 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7522c55f6438d270d73469d5fec09f76de9bbb452ef58166233fbb310c00725f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7522c55f6438d270d73469d5fec09f76de9bbb452ef58166233fbb310c00725f/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7de20525aa734c869cf50481e66383dba843e1abc1346e0984414ac65ba8d0dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7de20525aa734c869cf50481e66383dba843e1abc1346e0984414ac65ba8d0dc/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80027e2d8d189398ede870f89b1531f21dc99a0fdf39423eabc08a2f653f698a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80027e2d8d189398ede870f89b1531f21dc99a0fdf39423eabc08a2f653f698a/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9620b8e282fcd830eb914c216fa909a06260597ea5734be008fcfc0fe17e10e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9620b8e282fcd830eb914c216fa909a06260597ea5734be008fcfc0fe17e10e1/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a466dd4e385f3acf96c0d1cffe5a1abf0e239fe69b6417bbf62327a7b9a66cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a466dd4e385f3acf96c0d1cffe5a1abf0e239fe69b6417bbf62327a7b9a66cf/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a049d6608982bef0d437cb3ab400209dede225f9402b482ecb8347b30217549e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a049d6608982bef0d437cb3ab400209dede225f9402b482ecb8347b30217549e/userdata/shm major:0 minor:354 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4f7a821463b65fd299a7fdd03ef6a8cadd5ddeb430028d1e512a8d8c69dc117/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4f7a821463b65fd299a7fdd03ef6a8cadd5ddeb430028d1e512a8d8c69dc117/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aff104ed3eabbda7a26979c42d4f035cc30f6e99b4a2197a14e2014b3c91c1e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aff104ed3eabbda7a26979c42d4f035cc30f6e99b4a2197a14e2014b3c91c1e2/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3bc865020f000a0564451e1037d99663f52b28218d998575b70fa808077b428/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3bc865020f000a0564451e1037d99663f52b28218d998575b70fa808077b428/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5ccbc0b80c91de3303cbf85baaa8e4ef8c73804e35bd2fc88396217f79adeda/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5ccbc0b80c91de3303cbf85baaa8e4ef8c73804e35bd2fc88396217f79adeda/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e140f3f9baecf10768662ff7b249a07618d4ac75cd3a24775af125ba8a66caff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e140f3f9baecf10768662ff7b249a07618d4ac75cd3a24775af125ba8a66caff/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e21bfbc1e9afbf9cd3a266677b0bd6c791030049a7071b4b81631715f0425b1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e21bfbc1e9afbf9cd3a266677b0bd6c791030049a7071b4b81631715f0425b1d/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6af7c60346a713e8a6887b47105e85cf45efcebfd0f257edcc8cdf6bff15154/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6af7c60346a713e8a6887b47105e85cf45efcebfd0f257edcc8cdf6bff15154/userdata/shm major:0 minor:328 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7c95aa9c9172a1e1e8603fc2173aeef5ad3b11c79e3bcf8254e50b01236a132/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7c95aa9c9172a1e1e8603fc2173aeef5ad3b11c79e3bcf8254e50b01236a132/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa/volumes/kubernetes.io~projected/kube-api-access-fjp6p:{mountpoint:/var/lib/kubelet/pods/067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa/volumes/kubernetes.io~projected/kube-api-access-fjp6p major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa/volumes/kubernetes.io~secret/serving-cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06a672d7-0667-4bab-98ee-91dd71dfc489/volumes/kubernetes.io~projected/kube-api-access-9jrqs:{mountpoint:/var/lib/kubelet/pods/06a672d7-0667-4bab-98ee-91dd71dfc489/volumes/kubernetes.io~projected/kube-api-access-9jrqs major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1db07cc2-aa71-490e-a21a-0a8cb06d94d9/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/1db07cc2-aa71-490e-a21a-0a8cb06d94d9/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1db07cc2-aa71-490e-a21a-0a8cb06d94d9/volumes/kubernetes.io~projected/kube-api-access-lp68p:{mountpoint:/var/lib/kubelet/pods/1db07cc2-aa71-490e-a21a-0a8cb06d94d9/volumes/kubernetes.io~projected/kube-api-access-lp68p major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1db07cc2-aa71-490e-a21a-0a8cb06d94d9/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/1db07cc2-aa71-490e-a21a-0a8cb06d94d9/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1db07cc2-aa71-490e-a21a-0a8cb06d94d9/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/1db07cc2-aa71-490e-a21a-0a8cb06d94d9/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2000161a-2aa8-4638-a4f7-fbe305e929ae/volumes/kubernetes.io~projected/kube-api-access-9jchm:{mountpoint:/var/lib/kubelet/pods/2000161a-2aa8-4638-a4f7-fbe305e929ae/volumes/kubernetes.io~projected/kube-api-access-9jchm major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23c56cf7-2a99-4df8-841a-d84b6a9a3822/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/23c56cf7-2a99-4df8-841a-d84b6a9a3822/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23c56cf7-2a99-4df8-841a-d84b6a9a3822/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/23c56cf7-2a99-4df8-841a-d84b6a9a3822/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23c56cf7-2a99-4df8-841a-d84b6a9a3822/volumes/kubernetes.io~projected/kube-api-access-kt2rs:{mountpoint:/var/lib/kubelet/pods/23c56cf7-2a99-4df8-841a-d84b6a9a3822/volumes/kubernetes.io~projected/kube-api-access-kt2rs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35650ff5-011e-4572-886f-8bf47f598219/volumes/kubernetes.io~projected/kube-api-access-gvsxw:{mountpoint:/var/lib/kubelet/pods/35650ff5-011e-4572-886f-8bf47f598219/volumes/kubernetes.io~projected/kube-api-access-gvsxw major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/356632db-499f-4ced-bf01-b280472b53f1/volumes/kubernetes.io~projected/kube-api-access-pxb9x:{mountpoint:/var/lib/kubelet/pods/356632db-499f-4ced-bf01-b280472b53f1/volumes/kubernetes.io~projected/kube-api-access-pxb9x major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d34868b-605a-4bfb-81a3-432e0db0ce6a/volumes/kubernetes.io~projected/kube-api-access-wxp6s:{mountpoint:/var/lib/kubelet/pods/3d34868b-605a-4bfb-81a3-432e0db0ce6a/volumes/kubernetes.io~projected/kube-api-access-wxp6s major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/440dc059-df07-4f3e-a595-50d35d172e8c/volumes/kubernetes.io~projected/kube-api-access-f4hxv:{mountpoint:/var/lib/kubelet/pods/440dc059-df07-4f3e-a595-50d35d172e8c/volumes/kubernetes.io~projected/kube-api-access-f4hxv major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/440dc059-df07-4f3e-a595-50d35d172e8c/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/440dc059-df07-4f3e-a595-50d35d172e8c/volumes/kubernetes.io~secret/signing-key major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b84ba29-a116-48ef-8d9f-d81ceb4b3359/volumes/kubernetes.io~projected/kube-api-access-2p2xx:{mountpoint:/var/lib/kubelet/pods/4b84ba29-a116-48ef-8d9f-d81ceb4b3359/volumes/kubernetes.io~projected/kube-api-access-2p2xx major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52709fc1-b443-4a46-a95e-20cacc4ac4a0/volumes/kubernetes.io~projected/kube-api-access-qgjr7:{mountpoint:/var/lib/kubelet/pods/52709fc1-b443-4a46-a95e-20cacc4ac4a0/volumes/kubernetes.io~projected/kube-api-access-qgjr7 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cacf547-c691-4b14-936c-e6b08691bf93/volumes/kubernetes.io~projected/kube-api-access-fphb6:{mountpoint:/var/lib/kubelet/pods/5cacf547-c691-4b14-936c-e6b08691bf93/volumes/kubernetes.io~projected/kube-api-access-fphb6 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cacf547-c691-4b14-936c-e6b08691bf93/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/5cacf547-c691-4b14-936c-e6b08691bf93/volumes/kubernetes.io~secret/serving-cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6547c71f-71a2-40c9-912f-9283b6bdb560/volumes/kubernetes.io~projected/kube-api-access-qbv62:{mountpoint:/var/lib/kubelet/pods/6547c71f-71a2-40c9-912f-9283b6bdb560/volumes/kubernetes.io~projected/kube-api-access-qbv62 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6547c71f-71a2-40c9-912f-9283b6bdb560/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/6547c71f-71a2-40c9-912f-9283b6bdb560/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/784231bb-189f-472c-83ed-6ced092f10a1/volumes/kubernetes.io~projected/kube-api-access-nb9kd:{mountpoint:/var/lib/kubelet/pods/784231bb-189f-472c-83ed-6ced092f10a1/volumes/kubernetes.io~projected/kube-api-access-nb9kd major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ccca940-de12-451b-b920-d6bef516942f/volumes/kubernetes.io~projected/kube-api-access-fs262:{mountpoint:/var/lib/kubelet/pods/7ccca940-de12-451b-b920-d6bef516942f/volumes/kubernetes.io~projected/kube-api-access-fs262 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ccca940-de12-451b-b920-d6bef516942f/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/7ccca940-de12-451b-b920-d6bef516942f/volumes/kubernetes.io~secret/default-certificate major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ccca940-de12-451b-b920-d6bef516942f/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/7ccca940-de12-451b-b920-d6bef516942f/volumes/kubernetes.io~secret/stats-auth major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9046c34b-bb8f-4397-8b9d-7e2095add002/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/9046c34b-bb8f-4397-8b9d-7e2095add002/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/974ed564-6386-4e1e-9326-f14ccc36658c/volumes/kubernetes.io~projected/kube-api-access-4zn62:{mountpoint:/var/lib/kubelet/pods/974ed564-6386-4e1e-9326-f14ccc36658c/volumes/kubernetes.io~projected/kube-api-access-4zn62 major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a295913f-3811-47d6-9e85-a76182e45649/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/a295913f-3811-47d6-9e85-a76182e45649/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a295913f-3811-47d6-9e85-a76182e45649/volumes/kubernetes.io~projected/kube-api-access-42d4m:{mountpoint:/var/lib/kubelet/pods/a295913f-3811-47d6-9e85-a76182e45649/volumes/kubernetes.io~projected/kube-api-access-42d4m major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a295913f-3811-47d6-9e85-a76182e45649/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/a295913f-3811-47d6-9e85-a76182e45649/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aadb342f-3310-4c81-967d-0f0a8ef5a4e6/volumes/kubernetes.io~projected/kube-api-access-pv8v4:{mountpoint:/var/lib/kubelet/pods/aadb342f-3310-4c81-967d-0f0a8ef5a4e6/volumes/kubernetes.io~projected/kube-api-access-pv8v4 major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abf1a25c-1da1-449d-90c0-35d6fc406647/volumes/kubernetes.io~projected/kube-api-access-tv52h:{mountpoint:/var/lib/kubelet/pods/abf1a25c-1da1-449d-90c0-35d6fc406647/volumes/kubernetes.io~projected/kube-api-access-tv52h major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae204c86-8d93-4adf-988c-cd836e011270/volumes/kubernetes.io~projected/kube-api-access-hcxtm:{mountpoint:/var/lib/kubelet/pods/ae204c86-8d93-4adf-988c-cd836e011270/volumes/kubernetes.io~projected/kube-api-access-hcxtm major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbd3c724-e7e5-4382-b681-adf7fd8f302b/volumes/kubernetes.io~projected/kube-api-access-42cwb:{mountpoint:/var/lib/kubelet/pods/bbd3c724-e7e5-4382-b681-adf7fd8f302b/volumes/kubernetes.io~projected/kube-api-access-42cwb major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c27ff312-c8d3-4691-b535-ea8adf44d9c3/volumes/kubernetes.io~projected/kube-api-access-zpzsk:{mountpoint:/var/lib/kubelet/pods/c27ff312-c8d3-4691-b535-ea8adf44d9c3/volumes/kubernetes.io~projected/kube-api-access-zpzsk major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c27ff312-c8d3-4691-b535-ea8adf44d9c3/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/c27ff312-c8d3-4691-b535-ea8adf44d9c3/volumes/kubernetes.io~secret/cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3560fa9-2666-4e0d-854c-3c126489c61d/volumes/kubernetes.io~projected/kube-api-access-l6fs8:{mountpoint:/var/lib/kubelet/pods/c3560fa9-2666-4e0d-854c-3c126489c61d/volumes/kubernetes.io~projected/kube-api-access-l6fs8 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/caaa9f42-f8a2-4459-8917-a53e8fd852cc/volumes/kubernetes.io~projected/kube-api-access-mddbv:{mountpoint:/var/lib/kubelet/pods/caaa9f42-f8a2-4459-8917-a53e8fd852cc/volumes/kubernetes.io~projected/kube-api-access-mddbv major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cad3df5d-3fe9-449b-8b70-793bceb47985/volumes/kubernetes.io~projected/kube-api-access-wc77l:{mountpoint:/var/lib/kubelet/pods/cad3df5d-3fe9-449b-8b70-793bceb47985/volumes/kubernetes.io~projected/kube-api-access-wc77l major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c/volumes/kubernetes.io~projected/kube-api-access-fzr7h:{mountpoint:/var/lib/kubelet/pods/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c/volumes/kubernetes.io~projected/kube-api-access-fzr7h major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e703c9a2-a51a-4de6-b30c-6cc2aa810e57/volumes/kubernetes.io~projected/kube-api-access-dhskl:{mountpoint:/var/lib/kubelet/pods/e703c9a2-a51a-4de6-b30c-6cc2aa810e57/volumes/kubernetes.io~projected/kube-api-access-dhskl major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eea6e5c2-57f8-42c6-9ce9-55a3c8f92790/volumes/kubernetes.io~projected/kube-api-access-m22kb:{mountpoint:/var/lib/kubelet/pods/eea6e5c2-57f8-42c6-9ce9-55a3c8f92790/volumes/kubernetes.io~projected/kube-api-access-m22kb major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eea6e5c2-57f8-42c6-9ce9-55a3c8f92790/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/eea6e5c2-57f8-42c6-9ce9-55a3c8f92790/volumes/kubernetes.io~secret/serving-cert major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef7a859e-70f5-497a-9ad4-6614885fb71e/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/ef7a859e-70f5-497a-9ad4-6614885fb71e/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef7a859e-70f5-497a-9ad4-6614885fb71e/volumes/kubernetes.io~projected/kube-api-access-bhdwv:{mountpoint:/var/lib/kubelet/pods/ef7a859e-70f5-497a-9ad4-6614885fb71e/volumes/kubernetes.io~projected/kube-api-access-bhdwv major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef7a859e-70f5-497a-9ad4-6614885fb71e/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/ef7a859e-70f5-497a-9ad4-6614885fb71e/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef7a859e-70f5-497a-9ad4-6614885fb71e/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/ef7a859e-70f5-497a-9ad4-6614885fb71e/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f63c6326-4f32-4fa3-925a-83743ef7ff39/volumes/kubernetes.io~projected/kube-api-access-q7qxh:{mountpoint:/var/lib/kubelet/pods/f63c6326-4f32-4fa3-925a-83743ef7ff39/volumes/kubernetes.io~projected/kube-api-access-q7qxh major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6/volumes/kubernetes.io~projected/kube-api-access-m4vrx:{mountpoint:/var/lib/kubelet/pods/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6/volumes/kubernetes.io~projected/kube-api-access-m4vrx major:0 minor:61 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/58fd4a989a8c74ef9db5015373f731b5aa4ebb8c99eb7db4ef303b64c6b1be31/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/b5900356b6b14fd47a37a457c822e377f527d3d3290e51ed2e79cf7a080dd2a0/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/04915fd1013c7ea84769ddd741019c9fda583e58811b38919f9c1811a10f5e91/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/c76dd7478e066bcf5ff61738e333d1354a54af49307476755572071883564ffc/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/4e5c4a2669762a10b757d867ba89118e9778da9378482cfa79ab8a5ab8ecb515/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/9793d56bb1b7f275ff4dd4bd012fe6be5b635f51e5031542192ebfd6d7095e89/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/765901bf0676c02cf32616317842b24f506d2bb1b85c940a531852ed5bdd0c31/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/483fc10ed10e7bbd9f0f8ee94ca9216e75538594b3b616023cbd7a687be50ffc/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/f996dc253a9d28d6dd30e7eefe34d417c67c574f898f0b5c7eeee4182c4346e6/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/a894973035fa64b57ff42f014f0561c74f3bb7fb88e2f3e445b1a7643c270817/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/1329afc2ed8e5abfb9b8ca59d5e3073bfea833d2dd5532d9acc8bd8e2bc39952/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/ad6f0adb7c1e42030c209495fddad981874e8525f9b2ba99e47b2fdef38577a1/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/fdd18ade1aaada31f49ef08695231f1e8a37cfe8f430e76a8505b636c3c7ed3d/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/74aa3f8e6941d8f3b3209e575c0f87a33f53c04a789379eb57947a80711eb854/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/8d6b5f74b1ff3b2a97822e76657ecaae08da3186b265988c1560e49df9990fa1/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/71925e42c0e92a4d88b652cc8bdbd2250092cf819e8bc6b4aea9c5207d12a944/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/a892801748d2ab12049707c6eb3604c195b3255b93da65524a639d7c0994060b/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/eee8745b5319c02537fcbc4c840a738e63ab96e1964a43bd8c2c6c04c03d58f9/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/a69c9ea6b4b37a72bf14209891d1df726733921c6995cecc6d79c4cd42ff9330/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/97d64534d6f4ffd16ab91ddd524df5cf26706979527eaf9eaf6b21d5d6fc77d3/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/249eba11e1066d4e705153a923fed09318dec4f853db246503cbca52d4c78e3f/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/34431aabe05f6ab308e9392fc237053f96add92ea82e866734f21b86d06a3f00/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/a50c31effb305800ec746614cd6d3ba321b1b66e3f48a740524ea2fe35da38d1/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/68bdae98539cabdd93ffa4fd655753ce80cf20930b84392ac753543a150b2ef0/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/c21b6283db7c118185cb6158d36f3e2c39aeb2da1766fec6448893e26779c659/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/f5591d1ca19abb5a664c77abbfa5fbf1af18e782ab93332254d5aa5c6c80d830/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/42ac40f857d2513c0e0f81cb39217e97d903dab1c5038c3e984eff5336a5e5f9/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/1d3c93ff6a6fe0651d886ea1b4b78367cc4d57ce9179829f054d6d0ecab08e9c/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/e39e737416d18303022195fbba16b184cd7d9ee50302ed36de35e2c8592bc7ff/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/265c8c26582636869bfb20d30ae0e2ff57cf1966410e37997735723dbc998f12/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/2b35e6a260961c063b6dc4491ef09a34a60b4558309207372b7e57e730a4bd0d/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/5337778c13174b13cdc1db022759c0e61887d8bf93071ded376e551052f5b292/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/d6aa056da1c371eae60eaa5646078a137b2e1172864becf875f40d4fefc288e0/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/efda25c49248833a6451d125bbcb62738e248baab16d9a13343f148d96741dc6/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/f0860f84363dae9e80e6d040616bd1cb8ddb69b60a50a325c2e9a1da9da6a4cb/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/e2e25134d129fee91d68bc5af49be2f2d94c90e0319f629ef74cce65b30d5947/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/233a14de8812c1607bdaccf1b6039982c2b7a37db0bda22c3855bb62a5cbbffd/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/96ba614673bb97da77a654994d9f3b67502bb48febdae5ca30dacaf8fce65ed3/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/f802745e7079b0f514c673c039bdfee8ecf5c4e357c617c937c67f433a1f28c5/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/686dd8a58c9a55a4337be1a15df2475abeb313bcd5be4f7909c3f581e049aa78/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/12cb57d68d414c9d9dffc9508c5110fc7a65e9e4b9b9e5f90b250e1d23c4122e/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/feeebd55962a2bbc94d3d772223a48d0e570c18bd0fcb2bee27c228b54b90806/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/1ffd912e3e5dec15673939e18485510467cab892350ad0be29bf7670718eec08/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/34606a999801f0c788c338afc302095cd6cd888f4d3c3c3a5f48aefb24afd006/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/ea760be58b98b47d3f9ab616e65994ced8d001d4b877d75938046fc9e1126f12/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/1cb6049985136233df888d4af993d641c7ff3e8621533fe2820a59c545e0cca2/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/50ac311741f11f6206b64899617f5c3c3f0eba6980d6e1e8a99a7f9201f9c6cb/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/b3350324290282555329f17e0bcc115c37eb59b3e0ef8f3783bfc6d739f0effc/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/8f96722f59ea42ffe617228ae9b804a10b7246c5f70e49a0f47198db91e53655/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/837c2be4dd6226eee25b79d44b4e3ccf44da9b303a49196eb417db470332c7ee/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/94f1a4b8724499cfd3df1d6192a29cabb27f91311563f940983e4944b310dbd3/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/44c9bbd9e2cd7323f978b03e3ccdd1d62274fe3a6937225c03d0147fbcbd0afe/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/2fa89c7cd6584432f1eeaa612261de987521b5bbb609786d9b0a2be49b2df3b4/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/c954711a64293fa401617c3eedb5e6ce97031d270a87d687fd0e8d7ffdb20cb7/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/c973a0e683ceef5326d2b7a72a0102ccd84f879bba90c213c18fc0ac0a604612/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/a8e91b905bea5f69fa20bccb30961f75dedbabaa965b987b4a87cb6806bb0933/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/6fba218d9861a4c2f63bd4f69b1919d293ce2c7df9e7a5905fe73b0904ee4783/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/c423822f735d4743a060d63c1837e88971018ff80a566810be5217d41c12e16d/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/047bec977cdbc5e1189e99a14b0aa93b810f37c358a1800a66aee97847c7798f/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/d314a37b314edb22aea0e10efdf1987d2e523edc6312f3feab2744e242a165de/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 21:14:45.577922 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.577266    5053 manager.go:217] Machine: {Timestamp:2026-11-03 21:14:45.576430647 +0000 UTC m=+0.070352524 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3130791 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bcb1fce5030842ca50888659c98d1 SystemUUID:ec2bcb1f-ce50-3084-2ca5-0888659c98d1 BootID:60e91c6b-2423-4717-bd4a-369f194e7ccb Filesystems:[{Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef7a859e-70f5-497a-9ad4-6614885fb71e/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:231 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9620b8e282fcd830eb914c216fa909a06260597ea5734be008fcfc0fe17e10e1/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c3560fa9-2666-4e0d-854c-3c126489c61d/volumes/kubernetes.io~projected/kube-api-access-l6fs8 DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2000161a-2aa8-4638-a4f7-fbe305e929ae/volumes/kubernetes.io~projected/kube-api-access-9jchm DeviceMajor:0 DeviceMinor:346 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ccca940-de12-451b-b920-d6bef516942f/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a466dd4e385f3acf96c0d1cffe5a1abf0e239fe69b6417bbf62327a7b9a66cf/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eea6e5c2-57f8-42c6-9ce9-55a3c8f92790/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:280 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3d34868b-605a-4bfb-81a3-432e0db0ce6a/volumes/kubernetes.io~projected/kube-api-access-wxp6s DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5cacf547-c691-4b14-936c-e6b08691bf93/volumes/kubernetes.io~projected/kube-api-access-fphb6 DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6547c71f-71a2-40c9-912f-9283b6bdb560/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4f7a821463b65fd299a7fdd03ef6a8cadd5ddeb430028d1e512a8d8c69dc117/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f63c6326-4f32-4fa3-925a-83743ef7ff39/volumes/kubernetes.io~projected/kube-api-access-q7qxh DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/974ed564-6386-4e1e-9326-f14ccc36658c/volumes/kubernetes.io~projected/kube-api-access-4zn62 DeviceMajor:0 DeviceMinor:205 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ef7a859e-70f5-497a-9ad4-6614885fb71e/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/440dc059-df07-4f3e-a595-50d35d172e8c/volumes/kubernetes.io~projected/kube-api-access-f4hxv DeviceMajor:0 DeviceMinor:361 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/4b84ba29-a116-48ef-8d9f-d81ceb4b3359/volumes/kubernetes.io~projected/kube-api-access-2p2xx DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1db07cc2-aa71-490e-a21a-0a8cb06d94d9/volumes/kubernetes.io~projected/kube-api-access-lp68p DeviceMajor:0 DeviceMinor:230 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e140f3f9baecf10768662ff7b249a07618d4ac75cd3a24775af125ba8a66caff/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a295913f-3811-47d6-9e85-a76182e45649/volumes/kubernetes.io~projected/kube-api-access-42d4m DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bbd3c724-e7e5-4382-b681-adf7fd8f302b/volumes/kubernetes.io~projected/kube-api-access-42cwb DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/eea6e5c2-57f8-42c6-9ce9-55a3c8f92790/volumes/kubernetes.io~projected/kube-api-access-m22kb DeviceMajor:0 DeviceMinor:229 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6547c71f-71a2-40c9-912f-9283b6bdb560/volumes/kubernetes.io~projected/kube-api-access-qbv62 DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1db07cc2-aa71-490e-a21a-0a8cb06d94d9/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/097b094bd2ef653693302fbc8d2d34b595702dd18f0ece22727dec3df5532e2e/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ae204c86-8d93-4adf-988c-cd836e011270/volumes/kubernetes.io~projected/kube-api-access-hcxtm DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/784231bb-189f-472c-83ed-6ced092f10a1/volumes/kubernetes.io~projected/kube-api-access-nb9kd DeviceMajor:0 DeviceMinor:291 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/23c56cf7-2a99-4df8-841a-d84b6a9a3822/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7de20525aa734c869cf50481e66383dba843e1abc1346e0984414ac65ba8d0dc/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e703c9a2-a51a-4de6-b30c-6cc2aa810e57/volumes/kubernetes.io~projected/kube-api-access-dhskl DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ccca940-de12-451b-b920-d6bef516942f/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/caaa9f42-f8a2-4459-8917-a53e8fd852cc/volumes/kubernetes.io~projected/kube-api-access-mddbv DeviceMajor:0 DeviceMinor:327 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a049d6608982bef0d437cb3ab400209dede225f9402b482ecb8347b30217549e/userdata/shm DeviceMajor:0 DeviceMinor:354 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c27ff312-c8d3-4691-b535-ea8adf44d9c3/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:216 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2671cefed3cb069b2b6da7234d7d5f384a14b35dd5a2ccbe39248b441f34666b/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/440dc059-df07-4f3e-a595-50d35d172e8c/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:360 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/5cacf547-c691-4b14-936c-e6b08691bf93/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa/volumes/kubernetes.io~projected/kube-api-access-fjp6p DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9046c34b-bb8f-4397-8b9d-7e2095add002/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/324ab33cc272046022d3836ad1f36a85253684b77101adc607fdacf44b143a2b/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e21bfbc1e9afbf9cd3a266677b0bd6c791030049a7071b4b81631715f0425b1d/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d3bc865020f000a0564451e1037d99663f52b28218d998575b70fa808077b428/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6af7c60346a713e8a6887b47105e85cf45efcebfd0f257edcc8cdf6bff15154/userdata/shm DeviceMajor:0 DeviceMinor:328 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a295913f-3811-47d6-9e85-a76182e45649/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6/volumes/kubernetes.io~projected/kube-api-access-m4vrx DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a15829a8bd8ad5d05c335ab39dbdf0c9aa529ac4f64fcb36679332864076283/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef7a859e-70f5-497a-9ad4-6614885fb71e/volumes/kubernetes.io~projected/kube-api-access-bhdwv DeviceMajor:0 DeviceMinor:228 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/66bc4eaa57d92441818ddc2ee8a2a59f5f8676389ea6efbe33c1732ea677be6f/userdata/shm DeviceMajor:0 DeviceMinor:352 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/35650ff5-011e-4572-886f-8bf47f598219/volumes/kubernetes.io~projected/kube-api-access-gvsxw DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/52709fc1-b443-4a46-a95e-20cacc4ac4a0/volumes/kubernetes.io~projected/kube-api-access-qgjr7 DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7ccca940-de12-451b-b920-d6bef516942f/volumes/kubernetes.io~projected/kube-api-access-fs262 DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/047235fabf11c01f841f93ff511e942d78c3561e32d9afb78b1ee150f1fcc226/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a295913f-3811-47d6-9e85-a76182e45649/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/aadb342f-3310-4c81-967d-0f0a8ef5a4e6/volumes/kubernetes.io~projected/kube-api-access-pv8v4 DeviceMajor:0 DeviceMinor:233 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/23c56cf7-2a99-4df8-841a-d84b6a9a3822/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/356632db-499f-4ced-bf01-b280472b53f1/volumes/kubernetes.io~projected/kube-api-access-pxb9x DeviceMajor:0 DeviceMinor:351 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/654d663557e8f82d99dcdd5803bd5874617cadba42096308eff8d1f02824feab/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80027e2d8d189398ede870f89b1531f21dc99a0fdf39423eabc08a2f653f698a/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c/volumes/kubernetes.io~projected/kube-api-access-fzr7h DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/39fa4173aebabde72c8d3f7f8b97eae39ea8e8495f4c98be0237f5a3f3e78f0c/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/abf1a25c-1da1-449d-90c0-35d6fc406647/volumes/kubernetes.io~projected/kube-api-access-tv52h DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1db07cc2-aa71-490e-a21a-0a8cb06d94d9/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:214 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aff104ed3eabbda7a26979c42d4f035cc30f6e99b4a2197a14e2014b3c91c1e2/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/23c56cf7-2a99-4df8-841a-d84b6a9a3822/volumes/kubernetes.io~projected/kube-api-access-kt2rs DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1db07cc2-aa71-490e-a21a-0a8cb06d94d9/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:235 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19c349dd462dcbcf83ae4a8b956193a1b9484ecd552fa9ed773e410c18d16d79/userdata/shm DeviceMajor:0 DeviceMinor:52 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2283513bde82fc74d844a3598f31a3ab9f6a0e1b59d1aab361f8cadf549d543b/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cad3df5d-3fe9-449b-8b70-793bceb47985/volumes/kubernetes.io~projected/kube-api-access-wc77l DeviceMajor:0 DeviceMinor:322 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7522c55f6438d270d73469d5fec09f76de9bbb452ef58166233fbb310c00725f/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/06a672d7-0667-4bab-98ee-91dd71dfc489/volumes/kubernetes.io~projected/kube-api-access-9jrqs DeviceMajor:0 DeviceMinor:232 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5ccbc0b80c91de3303cbf85baaa8e4ef8c73804e35bd2fc88396217f79adeda/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef7a859e-70f5-497a-9ad4-6614885fb71e/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:215 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7c95aa9c9172a1e1e8603fc2173aeef5ad3b11c79e3bcf8254e50b01236a132/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0f7c57ffd61c558ae200d75efe4615c846e43a8fbfc892cb5dec957acbfab481/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c27ff312-c8d3-4691-b535-ea8adf44d9c3/volumes/kubernetes.io~projected/kube-api-access-zpzsk DeviceMajor:0 DeviceMinor:234 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42ea59ba3e76bc7fd70fe4215d37b5fcaa705382189b846299d9f3659b476d02/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2283513bde82fc7 MacAddress:ea:3d:95:46:bf:f8 Speed:10000 Mtu:8901} {Name:2671cefed3cb069 MacAddress:ba:30:ea:43:0a:ee Speed:10000 Mtu:8901} {Name:39fa4173aebabde MacAddress:a2:42:56:d3:7f:8f Speed:10000 Mtu:8901} {Name:42ea59ba3e76bc7 MacAddress:16:f5:82:3e:67:cd Speed:10000 Mtu:8901} {Name:66bc4eaa57d9244 MacAddress:4e:13:eb:2c:8c:03 Speed:10000 Mtu:8901} {Name:7522c55f6438d27 MacAddress:42:a5:f7:11:36:21 Speed:10000 Mtu:8901} {Name:7de20525aa734c8 MacAddress:1e:a7:50:c4:e9:9b Speed:10000 Mtu:8901} {Name:9620b8e282fcd83 MacAddress:2e:0c:c7:0e:d8:42 Speed:10000 Mtu:8901} {Name:a049d6608982bef MacAddress:16:e1:e3:d2:1b:51 Speed:10000 Mtu:8901} {Name:a4f7a821463b65f MacAddress:16:11:fb:f9:bf:bd Speed:10000 Mtu:8901} {Name:aff104ed3eabbda MacAddress:4e:de:33:01:bd:37 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:0d:50:7a:09:f3 Speed:0 Mtu:9001} {Name:br-int MacAddress:26:58:2a:70:b0:13 Speed:0 Mtu:8901} {Name:d3bc865020f000a MacAddress:5a:e5:68:8e:e6:d3 Speed:10000 Mtu:8901} {Name:d5ccbc0b80c91de MacAddress:06:6b:02:2a:74:20 Speed:10000 Mtu:8901} {Name:e21bfbc1e9afbf9 MacAddress:22:e4:5e:a2:9c:8d Speed:10000 Mtu:8901} {Name:e6af7c60346a713 MacAddress:b6:48:84:e8:e6:a3 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:0d:50:7a:09:f3 Speed:0 Mtu:9001} {Name:f7c95aa9c9172a1 MacAddress:1e:02:20:22:40:78 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:02:cf:0f:10:92:73 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:6e:17:7f:68:38:23 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 21:14:45.578139 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.577934    5053 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 21:14:45.578139 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.578002    5053 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 21:14:45.578406 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.578380    5053 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 21:14:45.578572 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.578408    5053 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-43.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 21:14:45.578646 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.578575    5053 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 21:14:45.578646 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.578589    5053 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 21:14:45.578646 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.578603    5053 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 21:14:45.578646 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.578622    5053 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 21:14:45.578830 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.578649    5053 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 21:14:45.578990 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.578975    5053 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 21:14:45.579125 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.579114    5053 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 21:14:45.579180 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.579164    5053 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 21:14:45.579225 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.579181    5053 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 21:14:45.579225 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.579196    5053 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 21:14:45.579225 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.579212    5053 kubelet.go:405] "Adding apiserver pod source"
Nov 03 21:14:45.579370 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.579251    5053 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 21:14:45.581438 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.580809    5053 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 21:14:45.581438 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.580829    5053 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 21:14:45.581614 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.581575    5053 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 21:14:45.607119 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.607097    5053 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 21:14:45.607119 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.607115    5053 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 21:14:45.607290 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.607222    5053 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 21:14:45.607444 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.607433    5053 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 21:14:45.607474 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.607463    5053 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 21:14:45.607560 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.607552    5053 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 21:14:45.607595 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.607565    5053 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 21:14:45.607595 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.607572    5053 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 21:14:45.607595 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.607577    5053 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 21:14:45.607595 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.607583    5053 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 21:14:45.607595 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.607588    5053 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 21:14:45.607595 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.607594    5053 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 21:14:45.607802 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.607599    5053 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 21:14:45.607802 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.607606    5053 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 21:14:45.607802 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.607612    5053 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 21:14:45.607802 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.607621    5053 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 21:14:45.607802 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.607629    5053 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 21:14:45.607802 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.607649    5053 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 21:14:45.607802 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.607655    5053 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 21:14:45.611246 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.611220    5053 server.go:1263] "Started kubelet"
Nov 03 21:14:45.611539 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.611485    5053 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 21:14:45.611812 ip-10-0-128-43 systemd[1]: Started Kubernetes Kubelet.
Nov 03 21:14:45.612754 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.612736    5053 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 21:14:45.613427 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.611454    5053 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 21:14:45.613504 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.613449    5053 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 21:14:45.613759 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.613741    5053 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 21:14:45.614711 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.614691    5053 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 21:14:45.614963 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.614774    5053 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 21:14:45.615035 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.614964    5053 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 21:14:45.615035 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.614982    5053 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 21:14:45.615035 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.614994    5053 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 21:09:05 +0000 UTC" deadline="2028-04-08 17:16:00.056563866 +0000 UTC"
Nov 03 21:14:45.615035 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.615031    5053 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12524h1m14.441536615s"
Nov 03 21:14:45.615222 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.615080    5053 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 21:14:45.616063 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:45.615914    5053 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-43.us-west-1.compute.internal\" not found"
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618206    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="974ed564-6386-4e1e-9326-f14ccc36658c" volumeName="kubernetes.io/projected/974ed564-6386-4e1e-9326-f14ccc36658c-kube-api-access-4zn62" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618301    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aadb342f-3310-4c81-967d-0f0a8ef5a4e6" volumeName="kubernetes.io/projected/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-kube-api-access-pv8v4" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618319    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abf1a25c-1da1-449d-90c0-35d6fc406647" volumeName="kubernetes.io/configmap/abf1a25c-1da1-449d-90c0-35d6fc406647-cni-binary-copy" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618340    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb8a800c-36be-4da0-b1e6-5ef65f994fc1" volumeName="kubernetes.io/configmap/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-nginx-conf" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618354    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c27ff312-c8d3-4691-b535-ea8adf44d9c3" volumeName="kubernetes.io/secret/c27ff312-c8d3-4691-b535-ea8adf44d9c3-cert" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618372    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7a859e-70f5-497a-9ad4-6614885fb71e" volumeName="kubernetes.io/configmap/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-certificates" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618384    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2000161a-2aa8-4638-a4f7-fbe305e929ae" volumeName="kubernetes.io/projected/2000161a-2aa8-4638-a4f7-fbe305e929ae-kube-api-access-9jchm" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618396    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ccca940-de12-451b-b920-d6bef516942f" volumeName="kubernetes.io/projected/7ccca940-de12-451b-b920-d6bef516942f-kube-api-access-fs262" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618416    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbd3c724-e7e5-4382-b681-adf7fd8f302b" volumeName="kubernetes.io/projected/bbd3c724-e7e5-4382-b681-adf7fd8f302b-kube-api-access-42cwb" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618428    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cad3df5d-3fe9-449b-8b70-793bceb47985" volumeName="kubernetes.io/empty-dir/cad3df5d-3fe9-449b-8b70-793bceb47985-data-volume" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618446    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cad3df5d-3fe9-449b-8b70-793bceb47985" volumeName="kubernetes.io/projected/cad3df5d-3fe9-449b-8b70-793bceb47985-kube-api-access-wc77l" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618459    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7a859e-70f5-497a-9ad4-6614885fb71e" volumeName="kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-bound-sa-token" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618475    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a295913f-3811-47d6-9e85-a76182e45649" volumeName="kubernetes.io/configmap/a295913f-3811-47d6-9e85-a76182e45649-ovnkube-config" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618487    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e703c9a2-a51a-4de6-b30c-6cc2aa810e57" volumeName="kubernetes.io/configmap/e703c9a2-a51a-4de6-b30c-6cc2aa810e57-iptables-alerter-script" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618510    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eea6e5c2-57f8-42c6-9ce9-55a3c8f92790" volumeName="kubernetes.io/projected/eea6e5c2-57f8-42c6-9ce9-55a3c8f92790-kube-api-access-m22kb" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618522    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f63c6326-4f32-4fa3-925a-83743ef7ff39" volumeName="kubernetes.io/configmap/f63c6326-4f32-4fa3-925a-83743ef7ff39-cni-binary-copy" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618539    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cacf547-c691-4b14-936c-e6b08691bf93" volumeName="kubernetes.io/configmap/5cacf547-c691-4b14-936c-e6b08691bf93-service-ca-bundle" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618553    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cacf547-c691-4b14-936c-e6b08691bf93" volumeName="kubernetes.io/empty-dir/5cacf547-c691-4b14-936c-e6b08691bf93-tmp" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618565    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23c56cf7-2a99-4df8-841a-d84b6a9a3822" volumeName="kubernetes.io/empty-dir/23c56cf7-2a99-4df8-841a-d84b6a9a3822-tmp" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618580    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c27ff312-c8d3-4691-b535-ea8adf44d9c3" volumeName="kubernetes.io/projected/c27ff312-c8d3-4691-b535-ea8adf44d9c3-kube-api-access-zpzsk" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618593    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c" volumeName="kubernetes.io/configmap/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-config" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618610    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f63c6326-4f32-4fa3-925a-83743ef7ff39" volumeName="kubernetes.io/configmap/f63c6326-4f32-4fa3-925a-83743ef7ff39-multus-daemon-config" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618621    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f63c6326-4f32-4fa3-925a-83743ef7ff39" volumeName="kubernetes.io/projected/f63c6326-4f32-4fa3-925a-83743ef7ff39-kube-api-access-q7qxh" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618635    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a295913f-3811-47d6-9e85-a76182e45649" volumeName="kubernetes.io/configmap/a295913f-3811-47d6-9e85-a76182e45649-env-overrides" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618650    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aadb342f-3310-4c81-967d-0f0a8ef5a4e6" volumeName="kubernetes.io/configmap/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-config-volume" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618664    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1db07cc2-aa71-490e-a21a-0a8cb06d94d9" volumeName="kubernetes.io/secret/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-installation-pull-secrets" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618682    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1db07cc2-aa71-490e-a21a-0a8cb06d94d9" volumeName="kubernetes.io/secret/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-image-registry-private-configuration" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618694    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="356632db-499f-4ced-bf01-b280472b53f1" volumeName="kubernetes.io/projected/356632db-499f-4ced-bf01-b280472b53f1-kube-api-access-pxb9x" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618717    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="440dc059-df07-4f3e-a595-50d35d172e8c" volumeName="kubernetes.io/projected/440dc059-df07-4f3e-a595-50d35d172e8c-kube-api-access-f4hxv" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618731    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9046c34b-bb8f-4397-8b9d-7e2095add002" volumeName="kubernetes.io/secret/9046c34b-bb8f-4397-8b9d-7e2095add002-agent-certs" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618749    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caaa9f42-f8a2-4459-8917-a53e8fd852cc" volumeName="kubernetes.io/empty-dir/caaa9f42-f8a2-4459-8917-a53e8fd852cc-archives-path" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618829    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6547c71f-71a2-40c9-912f-9283b6bdb560" volumeName="kubernetes.io/configmap/6547c71f-71a2-40c9-912f-9283b6bdb560-config" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618841    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96fee653-58b4-46bf-9ed5-0d3c5a93e8a8" volumeName="kubernetes.io/secret/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-original-pull-secret" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618859    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aadb342f-3310-4c81-967d-0f0a8ef5a4e6" volumeName="kubernetes.io/empty-dir/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-tmp-dir" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618870    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ccca940-de12-451b-b920-d6bef516942f" volumeName="kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-default-certificate" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618889    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae204c86-8d93-4adf-988c-cd836e011270" volumeName="kubernetes.io/configmap/ae204c86-8d93-4adf-988c-cd836e011270-telemetry-config" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618901    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3560fa9-2666-4e0d-854c-3c126489c61d" volumeName="kubernetes.io/projected/c3560fa9-2666-4e0d-854c-3c126489c61d-kube-api-access-l6fs8" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618913    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e703c9a2-a51a-4de6-b30c-6cc2aa810e57" volumeName="kubernetes.io/projected/e703c9a2-a51a-4de6-b30c-6cc2aa810e57-kube-api-access-dhskl" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618929    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d34868b-605a-4bfb-81a3-432e0db0ce6a" volumeName="kubernetes.io/projected/3d34868b-605a-4bfb-81a3-432e0db0ce6a-kube-api-access-wxp6s" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618941    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cacf547-c691-4b14-936c-e6b08691bf93" volumeName="kubernetes.io/secret/5cacf547-c691-4b14-936c-e6b08691bf93-serving-cert" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618963    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ccca940-de12-451b-b920-d6bef516942f" volumeName="kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-stats-auth" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618975    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abf1a25c-1da1-449d-90c0-35d6fc406647" volumeName="kubernetes.io/configmap/abf1a25c-1da1-449d-90c0-35d6fc406647-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.618986    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abf1a25c-1da1-449d-90c0-35d6fc406647" volumeName="kubernetes.io/projected/abf1a25c-1da1-449d-90c0-35d6fc406647-kube-api-access-tv52h" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619002    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eea6e5c2-57f8-42c6-9ce9-55a3c8f92790" volumeName="kubernetes.io/configmap/eea6e5c2-57f8-42c6-9ce9-55a3c8f92790-config" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619013    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7a859e-70f5-497a-9ad4-6614885fb71e" volumeName="kubernetes.io/empty-dir/ef7a859e-70f5-497a-9ad4-6614885fb71e-ca-trust-extracted" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619033    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7a859e-70f5-497a-9ad4-6614885fb71e" volumeName="kubernetes.io/secret/ef7a859e-70f5-497a-9ad4-6614885fb71e-installation-pull-secrets" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619047    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06a672d7-0667-4bab-98ee-91dd71dfc489" volumeName="kubernetes.io/projected/06a672d7-0667-4bab-98ee-91dd71dfc489-kube-api-access-9jrqs" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619060    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d34868b-605a-4bfb-81a3-432e0db0ce6a" volumeName="kubernetes.io/empty-dir/3d34868b-605a-4bfb-81a3-432e0db0ce6a-tmp-dir" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619076    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cacf547-c691-4b14-936c-e6b08691bf93" volumeName="kubernetes.io/projected/5cacf547-c691-4b14-936c-e6b08691bf93-kube-api-access-fphb6" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619089    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="784231bb-189f-472c-83ed-6ced092f10a1" volumeName="kubernetes.io/projected/784231bb-189f-472c-83ed-6ced092f10a1-kube-api-access-nb9kd" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619105    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abf1a25c-1da1-449d-90c0-35d6fc406647" volumeName="kubernetes.io/configmap/abf1a25c-1da1-449d-90c0-35d6fc406647-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619149    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7a859e-70f5-497a-9ad4-6614885fb71e" volumeName="kubernetes.io/secret/ef7a859e-70f5-497a-9ad4-6614885fb71e-image-registry-private-configuration" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619160    5053 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619163    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f83a41bd-5f2e-4c82-a357-ddb5be4be2f6" volumeName="kubernetes.io/projected/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-kube-api-access-m4vrx" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619175    5053 factory.go:55] Registering systemd factory
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619181    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b84ba29-a116-48ef-8d9f-d81ceb4b3359" volumeName="kubernetes.io/projected/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-kube-api-access-2p2xx" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619195    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cacf547-c691-4b14-936c-e6b08691bf93" volumeName="kubernetes.io/empty-dir/5cacf547-c691-4b14-936c-e6b08691bf93-snapshots" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619211    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6547c71f-71a2-40c9-912f-9283b6bdb560" volumeName="kubernetes.io/configmap/6547c71f-71a2-40c9-912f-9283b6bdb560-trusted-ca" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619258    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caaa9f42-f8a2-4459-8917-a53e8fd852cc" volumeName="kubernetes.io/configmap/caaa9f42-f8a2-4459-8917-a53e8fd852cc-service-ca-bundle" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619277    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c" volumeName="kubernetes.io/projected/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-kube-api-access-fzr7h" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619293    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caaa9f42-f8a2-4459-8917-a53e8fd852cc" volumeName="kubernetes.io/projected/caaa9f42-f8a2-4459-8917-a53e8fd852cc-kube-api-access-mddbv" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619307    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a295913f-3811-47d6-9e85-a76182e45649" volumeName="kubernetes.io/secret/a295913f-3811-47d6-9e85-a76182e45649-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619325    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="440dc059-df07-4f3e-a595-50d35d172e8c" volumeName="kubernetes.io/secret/440dc059-df07-4f3e-a595-50d35d172e8c-signing-key" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619337    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52709fc1-b443-4a46-a95e-20cacc4ac4a0" volumeName="kubernetes.io/projected/52709fc1-b443-4a46-a95e-20cacc4ac4a0-kube-api-access-qgjr7" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619350    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa" volumeName="kubernetes.io/configmap/067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa-config" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619382    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa" volumeName="kubernetes.io/secret/067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa-serving-cert" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619395    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1db07cc2-aa71-490e-a21a-0a8cb06d94d9" volumeName="kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-bound-sa-token" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619413    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23c56cf7-2a99-4df8-841a-d84b6a9a3822" volumeName="kubernetes.io/empty-dir/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-tuned" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619424    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="440dc059-df07-4f3e-a595-50d35d172e8c" volumeName="kubernetes.io/configmap/440dc059-df07-4f3e-a595-50d35d172e8c-signing-cabundle" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619437    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1db07cc2-aa71-490e-a21a-0a8cb06d94d9" volumeName="kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-kube-api-access-lp68p" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619454    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3560fa9-2666-4e0d-854c-3c126489c61d" volumeName="kubernetes.io/configmap/c3560fa9-2666-4e0d-854c-3c126489c61d-serviceca" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619465    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7a859e-70f5-497a-9ad4-6614885fb71e" volumeName="kubernetes.io/configmap/ef7a859e-70f5-497a-9ad4-6614885fb71e-trusted-ca" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619481    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa" volumeName="kubernetes.io/projected/067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa-kube-api-access-fjp6p" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619493    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae204c86-8d93-4adf-988c-cd836e011270" volumeName="kubernetes.io/projected/ae204c86-8d93-4adf-988c-cd836e011270-kube-api-access-hcxtm" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619515    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1db07cc2-aa71-490e-a21a-0a8cb06d94d9" volumeName="kubernetes.io/configmap/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-trusted-ca" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619532    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1db07cc2-aa71-490e-a21a-0a8cb06d94d9" volumeName="kubernetes.io/empty-dir/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-ca-trust-extracted" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619543    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23c56cf7-2a99-4df8-841a-d84b6a9a3822" volumeName="kubernetes.io/projected/23c56cf7-2a99-4df8-841a-d84b6a9a3822-kube-api-access-kt2rs" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619560    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9046c34b-bb8f-4397-8b9d-7e2095add002" volumeName="kubernetes.io/configmap/9046c34b-bb8f-4397-8b9d-7e2095add002-konnectivity-ca" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619572    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a295913f-3811-47d6-9e85-a76182e45649" volumeName="kubernetes.io/projected/a295913f-3811-47d6-9e85-a76182e45649-kube-api-access-42d4m" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619588    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eea6e5c2-57f8-42c6-9ce9-55a3c8f92790" volumeName="kubernetes.io/secret/eea6e5c2-57f8-42c6-9ce9-55a3c8f92790-serving-cert" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619605    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1db07cc2-aa71-490e-a21a-0a8cb06d94d9" volumeName="kubernetes.io/configmap/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-certificates" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619630    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6547c71f-71a2-40c9-912f-9283b6bdb560" volumeName="kubernetes.io/projected/6547c71f-71a2-40c9-912f-9283b6bdb560-kube-api-access-qbv62" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619646    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6547c71f-71a2-40c9-912f-9283b6bdb560" volumeName="kubernetes.io/secret/6547c71f-71a2-40c9-912f-9283b6bdb560-serving-cert" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619658    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a295913f-3811-47d6-9e85-a76182e45649" volumeName="kubernetes.io/configmap/a295913f-3811-47d6-9e85-a76182e45649-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619669    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caaa9f42-f8a2-4459-8917-a53e8fd852cc" volumeName="kubernetes.io/configmap/caaa9f42-f8a2-4459-8917-a53e8fd852cc-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619686    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7a859e-70f5-497a-9ad4-6614885fb71e" volumeName="kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-kube-api-access-bhdwv" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619699    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cacf547-c691-4b14-936c-e6b08691bf93" volumeName="kubernetes.io/configmap/5cacf547-c691-4b14-936c-e6b08691bf93-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619715    5053 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35650ff5-011e-4572-886f-8bf47f598219" volumeName="kubernetes.io/projected/35650ff5-011e-4572-886f-8bf47f598219-kube-api-access-gvsxw" seLinuxMountContext=""
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619727    5053 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619735    5053 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 21:14:45.621366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.619185    5053 factory.go:223] Registration of the systemd container factory successfully
Nov 03 21:14:45.625404 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.622054    5053 factory.go:153] Registering CRI-O factory
Nov 03 21:14:45.625404 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.622074    5053 factory.go:223] Registration of the crio container factory successfully
Nov 03 21:14:45.625404 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.622098    5053 factory.go:103] Registering Raw factory
Nov 03 21:14:45.625404 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.622136    5053 manager.go:1199] Started watching for new ooms in manager
Nov 03 21:14:45.625404 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.622514    5053 manager.go:319] Starting recovery of all containers
Nov 03 21:14:45.633264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.633218    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/1.log"
Nov 03 21:14:45.633624 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.633585    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/1.log"
Nov 03 21:14:45.648179 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.648155    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:14:45.648624 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.648601    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:14:45.657068 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.657037    5053 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 21:14:45.658604 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.658578    5053 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 21:14:45.658604 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.658604    5053 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 21:14:45.658765 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.658631    5053 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 21:14:45.658765 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:45.658670    5053 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 21:14:45.666000 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.665973    5053 generic.go:356] "Generic (PLEG): container finished" podID="abf1a25c-1da1-449d-90c0-35d6fc406647" containerID="41a075e0982a2efef1cd88f6ae89ac3bc5581d050ba45031e232c03648799294" exitCode=0
Nov 03 21:14:45.666000 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.665997    5053 generic.go:356] "Generic (PLEG): container finished" podID="abf1a25c-1da1-449d-90c0-35d6fc406647" containerID="4fc59f451eac6beab6e223e05687434c39bbff6d9fb541a52c5ba5bf86ec28ac" exitCode=0
Nov 03 21:14:45.666156 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.666005    5053 generic.go:356] "Generic (PLEG): container finished" podID="abf1a25c-1da1-449d-90c0-35d6fc406647" containerID="3297410a43be62f39b61871bd6d6a8dd5aa0fff569ea925cab41f01be6d346f2" exitCode=0
Nov 03 21:14:45.666156 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.666013    5053 generic.go:356] "Generic (PLEG): container finished" podID="abf1a25c-1da1-449d-90c0-35d6fc406647" containerID="11af2e37c817ac42d2d9fc208d673eba62ea9f605ef2f2307a4126b3acd77601" exitCode=0
Nov 03 21:14:45.666156 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.666020    5053 generic.go:356] "Generic (PLEG): container finished" podID="abf1a25c-1da1-449d-90c0-35d6fc406647" containerID="272e72e384ca30edaa0952cbe619c2d5cdc9431d9926e09a61b62a5d9221045b" exitCode=0
Nov 03 21:14:45.666156 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.666027    5053 generic.go:356] "Generic (PLEG): container finished" podID="abf1a25c-1da1-449d-90c0-35d6fc406647" containerID="3046d10cfc825df3046d4a961c19bf219066078ea1b9753c9457424084e2fc3e" exitCode=0
Nov 03 21:14:45.695721 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.695688    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:14:45.696113 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.696080    5053 generic.go:356] "Generic (PLEG): container finished" podID="a295913f-3811-47d6-9e85-a76182e45649" containerID="79d3b0235001a3a1ab99925d06b325fcab0760565afa681654a8a0ed6b18e50c" exitCode=1
Nov 03 21:14:45.717882 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:45.717854    5053 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-43.us-west-1.compute.internal\" not found"
Nov 03 21:14:45.739090 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.738443    5053 generic.go:356] "Generic (PLEG): container finished" podID="104317fa8cfa5db2df29de73ade6b849" containerID="616f6bda8df912c55f680ef7b4ba29e03af421160c74f93e5f090568611268aa" exitCode=0
Nov 03 21:14:45.746998 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.746964    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/1.log"
Nov 03 21:14:45.747209 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.747001    5053 generic.go:356] "Generic (PLEG): container finished" podID="6547c71f-71a2-40c9-912f-9283b6bdb560" containerID="9f6b6dc724c5973a7c58dc058c00d7b81472be8fbb45b2fedb085ed90d0ec098" exitCode=255
Nov 03 21:14:45.753576 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.753558    5053 manager.go:324] Recovery completed
Nov 03 21:14:45.761371 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:45.761352    5053 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 21:14:45.786317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.786293    5053 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 21:14:45.788309 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.788288    5053 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 21:14:45.788372 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.788323    5053 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 21:14:45.788372 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.788339    5053 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 21:14:45.790638 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.790621    5053 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 21:14:45.797936 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.797916    5053 cpu_manager.go:226] "Starting" policy="none"
Nov 03 21:14:45.797936 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.797934    5053 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 21:14:45.798092 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.797954    5053 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 21:14:45.798150 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.798110    5053 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 21:14:45.798150 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.798124    5053 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 21:14:45.798271 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.798258    5053 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 21:14:45.798321 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.798270    5053 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 21:14:45.798321 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.798278    5053 policy_none.go:50] "Start"
Nov 03 21:14:45.798321 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.798293    5053 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 21:14:45.798321 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.798305    5053 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 21:14:45.798497 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.798470    5053 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 21:14:45.798497 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.798480    5053 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 21:14:45.798497 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.798488    5053 policy_none.go:44] "Start"
Nov 03 21:14:45.798752 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.798733    5053 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 21:14:45.799107 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.799092    5053 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 21:14:45.799382 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.799368    5053 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 21:14:45.799773 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.799760    5053 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 21:14:45.803727 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.803710    5053 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 21:14:45.803844 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.803835    5053 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 21:14:45.803920 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.803912    5053 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 21:14:45.804740 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.804708    5053 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 21:14:45.813924 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.813908    5053 manager.go:341] "Starting Device Plugin manager"
Nov 03 21:14:45.814025 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:45.813940    5053 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 21:14:45.814025 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.813953    5053 server.go:87] "Starting device plugin registration server"
Nov 03 21:14:45.814826 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.814635    5053 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 21:14:45.814826 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.814654    5053 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 21:14:45.815086 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.815068    5053 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 21:14:45.815185 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.815155    5053 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 21:14:45.815257 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.815194    5053 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 21:14:45.815257 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.815200    5053 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 21:14:45.815355 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.815284    5053 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T21:14:45.815185004Z","UUID":"53641a6c-1699-4c3b-acdc-129d968c8988","Handler":null,"Name":"","Endpoint":""}
Nov 03 21:14:45.817936 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:45.816856    5053 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 21:14:45.817936 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.816904    5053 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 21:14:45.817936 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.816925    5053 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 21:14:45.847988 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.847965    5053 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 21:14:45.915302 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.915273    5053 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 21:14:45.916438 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.916421    5053 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 21:14:45.916495 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.916455    5053 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 21:14:45.916495 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.916466    5053 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-43.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 21:14:45.916566 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.916505    5053 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:45.961803 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.961710    5053 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal"]
Nov 03 21:14:45.962411 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.962389    5053 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:45.962491 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.962431    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal" event={"ID":"8253f9611b0fc9c6ff64af8baa405814","Type":"ContainerStarted","Data":"c2db52cc75ad5d65a686d94d85d6221bb9d797177fbd32be105d50d77e5d77be"}
Nov 03 21:14:45.962491 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.962465    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal" event={"ID":"8253f9611b0fc9c6ff64af8baa405814","Type":"ContainerStarted","Data":"0a15829a8bd8ad5d05c335ab39dbdf0c9aa529ac4f64fcb36679332864076283"}
Nov 03 21:14:45.962491 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.962487    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal" event={"ID":"104317fa8cfa5db2df29de73ade6b849","Type":"ContainerStarted","Data":"4f6bdf9728a3f68fad69573e80101c16ce5d15295a368fb4a036da1cf93760f1"}
Nov 03 21:14:45.962618 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.962502    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal" event={"ID":"104317fa8cfa5db2df29de73ade6b849","Type":"ContainerDied","Data":"616f6bda8df912c55f680ef7b4ba29e03af421160c74f93e5f090568611268aa"}
Nov 03 21:14:45.962618 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.962516    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal" event={"ID":"104317fa8cfa5db2df29de73ade6b849","Type":"ContainerStarted","Data":"19c349dd462dcbcf83ae4a8b956193a1b9484ecd552fa9ed773e410c18d16d79"}
Nov 03 21:14:45.962678 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:45.962630    5053 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:46.021851 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.021824    5053 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 21:14:46.021960 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:46.021874    5053 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:46.022313 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.022300    5053 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 21:14:46.022411 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:46.022333    5053 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:46.023859 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.023844    5053 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:46.023913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.023894    5053 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:46.091844 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.091807    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8253f9611b0fc9c6ff64af8baa405814-config\") pod \"kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal\" (UID: \"8253f9611b0fc9c6ff64af8baa405814\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:46.091981 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.091847    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/104317fa8cfa5db2df29de73ade6b849-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal\" (UID: \"104317fa8cfa5db2df29de73ade6b849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:46.091981 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.091878    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/104317fa8cfa5db2df29de73ade6b849-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal\" (UID: \"104317fa8cfa5db2df29de73ade6b849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:46.192728 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.192691    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/104317fa8cfa5db2df29de73ade6b849-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal\" (UID: \"104317fa8cfa5db2df29de73ade6b849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:46.192728 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.192729    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/104317fa8cfa5db2df29de73ade6b849-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal\" (UID: \"104317fa8cfa5db2df29de73ade6b849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:46.192927 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.192755    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8253f9611b0fc9c6ff64af8baa405814-config\") pod \"kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal\" (UID: \"8253f9611b0fc9c6ff64af8baa405814\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:46.192927 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.192803    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8253f9611b0fc9c6ff64af8baa405814-config\") pod \"kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal\" (UID: \"8253f9611b0fc9c6ff64af8baa405814\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:46.192927 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.192807    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/104317fa8cfa5db2df29de73ade6b849-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal\" (UID: \"104317fa8cfa5db2df29de73ade6b849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:46.192927 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.192809    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/104317fa8cfa5db2df29de73ade6b849-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal\" (UID: \"104317fa8cfa5db2df29de73ade6b849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:46.580151 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.580117    5053 apiserver.go:52] "Watching apiserver"
Nov 03 21:14:46.733309 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.733273    5053 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 21:14:46.734142 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.734117    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8","openshift-multus/network-metrics-daemon-6xvgt","openshift-kube-storage-version-migrator/migrator-66df5d496b-jqkbh","openshift-monitoring/cluster-monitoring-operator-884576c57-jf555","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz","openshift-ingress/router-default-5ffbbbfcf7-99qbj","openshift-multus/multus-67blc","kube-system/kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nv9qq","openshift-cluster-node-tuning-operator/tuned-6zndc","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7","kube-system/konnectivity-agent-cccr5","openshift-insights/periodic-gathering-g82mg-mqwqf","openshift-multus/multus-additional-cni-plugins-f2pgd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal","kube-system/kas-connection-checker-685895c7c9-27lr2","openshift-network-diagnostics/network-check-target-8w5st","openshift-ovn-kubernetes/ovnkube-node-fvkn6","openshift-image-registry/image-registry-6df897dfcf-2b92c","openshift-image-registry/image-registry-75cb56744b-c6rsx","openshift-insights/insights-operator-7c4986d449-bc6rw","openshift-insights/insights-runtime-extractor-v9vxl","openshift-dns/node-resolver-554sk","kube-system/kas-connection-checker-685895c7c9-sds64","openshift-console-operator/console-operator-7cd56b8dbc-rhpjv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr","openshift-network-operator/iptables-alerter-km5mx","kube-system/kas-connection-checker-685895c7c9-bhpkz","openshift-image-registry/node-ca-z5s4n","openshift-service-ca/service-ca-5fffc97bdf-zf7tn","openshift-dns/dns-default-8p8hk","openshift-network-console/networking-console-plugin-86c478bb88-j2ksl","openshift-kube-storage-version-migrator/migrator-66df5d496b-v6qdv","openshift-ingress-canary/ingress-canary-m5s9b","openshift-network-diagnostics/network-check-source-6f78d59f99-sb5xl","kube-system/global-pull-secret-syncer-v5vrf"]
Nov 03 21:14:46.735630 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.735611    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:46.736326 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.736301    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"
Nov 03 21:14:46.737365 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.737345    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:46.741666 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.741650    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-j2ksl"
Nov 03 21:14:46.742051 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.742038    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:46.747454 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.747439    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:46.748400 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.748382    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:46.748867 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.748853    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:46.750120 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.750105    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:46.763700 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.763672    5053 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:46.766765 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.766740    5053 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:46.786763 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.786733    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 21:14:46.786763 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.786748    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 21:14:46.787027 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.787012    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 21:14:46.787253 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.787225    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 21:14:46.787850 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.787834    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 21:14:46.787850 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.787838    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 21:14:46.788000 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.787964    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 21:14:46.788074 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.788058    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 21:14:46.788200 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.788188    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 21:14:46.788269 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.788218    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 21:14:46.788593 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.788582    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 21:14:46.788689 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.788677    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 21:14:46.788871 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.788858    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 21:14:46.788968 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.788955    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 21:14:46.788996 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.788974    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 21:14:46.789276 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.789255    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 21:14:46.789750 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.789738    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 21:14:46.790119 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.790106    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 21:14:46.790588 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.790571    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 21:14:46.790752 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.790641    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 21:14:46.817079 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.817057    5053 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 21:14:46.837528 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.837459    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 21:14:46.837528 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.837494    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 21:14:46.837669 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.837528    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 21:14:46.837669 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.837548    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 21:14:46.837669 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.837534    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 21:14:46.837669 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.837574    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 21:14:46.837669 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.837575    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 21:14:46.837834 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.837689    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 21:14:46.844412 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.844397    5053 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 21:14:46.844467 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:46.844439    5053 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:46.844700 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.844680    5053 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 21:14:46.844792 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:46.844724    5053 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-43.us-west-1.compute.internal"
Nov 03 21:14:46.851926 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.851904    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 21:14:46.852270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.852251    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vbwk2\""
Nov 03 21:14:46.852356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.852269    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6d8l7\""
Nov 03 21:14:46.852356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.852288    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 21:14:46.852356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.852262    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 03 21:14:46.852494 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.852385    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 21:14:46.852555 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.852539    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 21:14:46.852604 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.852587    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 21:14:46.852664 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.852601    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zzfft\""
Nov 03 21:14:46.852714 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.852682    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l78m6\""
Nov 03 21:14:46.852786 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.852770    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zbt6z\""
Nov 03 21:14:46.852890 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.852877    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4f9ng\""
Nov 03 21:14:46.853039 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.853021    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-trdz7\""
Nov 03 21:14:46.853104 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.853064    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rbksq\""
Nov 03 21:14:46.853104 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.853091    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-b97xq\""
Nov 03 21:14:46.853194 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.853156    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-smlpl\""
Nov 03 21:14:46.853343 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.853329    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w4rzx\""
Nov 03 21:14:46.853431 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.853392    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 21:14:46.853431 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.853422    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 21:14:46.853516 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.853449    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jnct2\""
Nov 03 21:14:46.853516 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.853451    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 21:14:46.853516 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.853509    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8zbhm\""
Nov 03 21:14:46.853630 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.853580    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jl8l5\""
Nov 03 21:14:46.853677 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.853657    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 21:14:46.853677 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.853660    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-l8m5w\""
Nov 03 21:14:46.853735 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.853705    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tq7pf\""
Nov 03 21:14:46.853768 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.853763    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 21:14:46.853814 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.853802    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 21:14:46.853903 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.853891    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cldzt\""
Nov 03 21:14:46.853978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.853967    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jxfhg\""
Nov 03 21:14:46.854068 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.854058    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 21:14:46.854100 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.854073    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-25rf5\""
Nov 03 21:14:46.854164 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.854123    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 21:14:46.854193 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.854169    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 21:14:46.854220 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.854211    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-md6qp\""
Nov 03 21:14:46.854314 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.854303    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lzq4j\""
Nov 03 21:14:46.886978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.886949    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 21:14:46.886978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.886968    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 21:14:46.887181 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.887133    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 21:14:46.887391 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.887376    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 21:14:46.887479 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.887468    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 21:14:46.887581 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.887570    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 21:14:46.887638 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.887627    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 21:14:46.887686 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.887629    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 21:14:46.887786 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.887769    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 21:14:46.887836 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.887810    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 21:14:46.887836 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.887818    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 21:14:46.888053 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888034    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 21:14:46.888107 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888037    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 21:14:46.888107 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888067    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 21:14:46.888107 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888092    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 21:14:46.888107 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888095    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 21:14:46.888268 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888122    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 21:14:46.888268 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888169    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 21:14:46.888268 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888169    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 21:14:46.888268 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888216    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 21:14:46.888268 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888267    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 21:14:46.888485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888292    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 21:14:46.888485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888307    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 21:14:46.888485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888358    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 21:14:46.888485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888399    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 21:14:46.888485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888419    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 21:14:46.888485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888426    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 21:14:46.888485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888437    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 21:14:46.888485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888402    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 21:14:46.888485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888427    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 21:14:46.888485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888460    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 21:14:46.888854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888554    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 21:14:46.888854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888563    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 21:14:46.888854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888617    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 21:14:46.888854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888625    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 21:14:46.888854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888617    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 21:14:46.888854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888654    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 21:14:46.888854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888666    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 21:14:46.888854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888731    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 21:14:46.888854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888742    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 21:14:46.889157 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.888968    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 21:14:46.896352 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.896336    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-installation-pull-secrets\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:46.896401 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.896364    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abf1a25c-1da1-449d-90c0-35d6fc406647-system-cni-dir\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:46.896401 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.896383    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/abf1a25c-1da1-449d-90c0-35d6fc406647-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:46.896560 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.896406    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/440dc059-df07-4f3e-a595-50d35d172e8c-signing-cabundle\") pod \"service-ca-5fffc97bdf-zf7tn\" (UID: \"440dc059-df07-4f3e-a595-50d35d172e8c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zf7tn"
Nov 03 21:14:46.896560 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.896435    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb9kd\" (UniqueName: \"kubernetes.io/projected/784231bb-189f-472c-83ed-6ced092f10a1-kube-api-access-nb9kd\") pod \"network-check-target-8w5st\" (UID: \"784231bb-189f-472c-83ed-6ced092f10a1\") " pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:46.896560 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.896469    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c3560fa9-2666-4e0d-854c-3c126489c61d-host\") pod \"node-ca-z5s4n\" (UID: \"c3560fa9-2666-4e0d-854c-3c126489c61d\") " pod="openshift-image-registry/node-ca-z5s4n"
Nov 03 21:14:46.896560 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.896525    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m22kb\" (UniqueName: \"kubernetes.io/projected/eea6e5c2-57f8-42c6-9ce9-55a3c8f92790-kube-api-access-m22kb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7\" (UID: \"eea6e5c2-57f8-42c6-9ce9-55a3c8f92790\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7"
Nov 03 21:14:46.896738 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.896578    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cacf547-c691-4b14-936c-e6b08691bf93-service-ca-bundle\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:46.896738 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.896607    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4vrx\" (UniqueName: \"kubernetes.io/projected/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-kube-api-access-m4vrx\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:46.896738 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.896633    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9046c34b-bb8f-4397-8b9d-7e2095add002-agent-certs\") pod \"konnectivity-agent-cccr5\" (UID: \"9046c34b-bb8f-4397-8b9d-7e2095add002\") " pod="kube-system/konnectivity-agent-cccr5"
Nov 03 21:14:46.896738 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.896660    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-run-multus-certs\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:46.896738 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.896687    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6547c71f-71a2-40c9-912f-9283b6bdb560-trusted-ca\") pod \"console-operator-7cd56b8dbc-rhpjv\" (UID: \"6547c71f-71a2-40c9-912f-9283b6bdb560\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:46.896738 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.896712    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/abf1a25c-1da1-449d-90c0-35d6fc406647-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:46.896738 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.896715    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvsxw\" (UniqueName: \"kubernetes.io/projected/35650ff5-011e-4572-886f-8bf47f598219-kube-api-access-gvsxw\") pod \"volume-data-source-validator-58c9f5f46d-nv9qq\" (UID: \"35650ff5-011e-4572-886f-8bf47f598219\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nv9qq"
Nov 03 21:14:46.897060 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.896714    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/440dc059-df07-4f3e-a595-50d35d172e8c-signing-cabundle\") pod \"service-ca-5fffc97bdf-zf7tn\" (UID: \"440dc059-df07-4f3e-a595-50d35d172e8c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zf7tn"
Nov 03 21:14:46.897060 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.896757    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-certificates\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:46.897060 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.896790    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/440dc059-df07-4f3e-a595-50d35d172e8c-signing-key\") pod \"service-ca-5fffc97bdf-zf7tn\" (UID: \"440dc059-df07-4f3e-a595-50d35d172e8c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zf7tn"
Nov 03 21:14:46.897060 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.896828    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cacf547-c691-4b14-936c-e6b08691bf93-service-ca-bundle\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:46.897060 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.896830    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-var-lib-kubelet\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:46.897060 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.896829    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-installation-pull-secrets\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:46.897060 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.896881    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-multus-conf-dir\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:46.897060 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.896924    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-device-dir\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:46.897060 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.896976    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-bound-sa-token\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:46.897060 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.897005    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jchm\" (UniqueName: \"kubernetes.io/projected/2000161a-2aa8-4638-a4f7-fbe305e929ae-kube-api-access-9jchm\") pod \"migrator-66df5d496b-jqkbh\" (UID: \"2000161a-2aa8-4638-a4f7-fbe305e929ae\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jqkbh"
Nov 03 21:14:46.897060 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.897032    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-var-lib-cni-bin\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:46.897060 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.897029    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6547c71f-71a2-40c9-912f-9283b6bdb560-trusted-ca\") pod \"console-operator-7cd56b8dbc-rhpjv\" (UID: \"6547c71f-71a2-40c9-912f-9283b6bdb560\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:46.897571 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.897105    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/440dc059-df07-4f3e-a595-50d35d172e8c-signing-key\") pod \"service-ca-5fffc97bdf-zf7tn\" (UID: \"440dc059-df07-4f3e-a595-50d35d172e8c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zf7tn"
Nov 03 21:14:46.897571 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.897154    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-original-pull-secret\") pod \"global-pull-secret-syncer-v5vrf\" (UID: \"96fee653-58b4-46bf-9ed5-0d3c5a93e8a8\") " pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:46.897571 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.897171    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9046c34b-bb8f-4397-8b9d-7e2095add002-agent-certs\") pod \"konnectivity-agent-cccr5\" (UID: \"9046c34b-bb8f-4397-8b9d-7e2095add002\") " pod="kube-system/konnectivity-agent-cccr5"
Nov 03 21:14:46.897571 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.897268    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:46.897571 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.897391    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-multus-socket-dir-parent\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:46.897571 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.897470    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.897571 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.897521    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-run\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:46.897571 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.897527    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-original-pull-secret\") pod \"global-pull-secret-syncer-v5vrf\" (UID: \"96fee653-58b4-46bf-9ed5-0d3c5a93e8a8\") " pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:46.897878 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.897606    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs\") pod \"network-metrics-daemon-6xvgt\" (UID: \"4b84ba29-a116-48ef-8d9f-d81ceb4b3359\") " pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:46.897878 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.897633    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ef7a859e-70f5-497a-9ad4-6614885fb71e-ca-trust-extracted\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:46.897878 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.897650    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-etc-kubernetes\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:46.897878 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.897666    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-certificates\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:46.897878 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.897670    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ae204c86-8d93-4adf-988c-cd836e011270-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jf555\" (UID: \"ae204c86-8d93-4adf-988c-cd836e011270\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:46.897878 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.897728    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ef7a859e-70f5-497a-9ad4-6614885fb71e-ca-trust-extracted\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:46.897878 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.897769    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eea6e5c2-57f8-42c6-9ce9-55a3c8f92790-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7\" (UID: \"eea6e5c2-57f8-42c6-9ce9-55a3c8f92790\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7"
Nov 03 21:14:46.897878 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.897801    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:46.897878 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.897847    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-multus-cni-dir\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:46.898332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.897884    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j2ksl\" (UID: \"bb8a800c-36be-4da0-b1e6-5ef65f994fc1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j2ksl"
Nov 03 21:14:46.898332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.897918    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a295913f-3811-47d6-9e85-a76182e45649-env-overrides\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.898332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.897932    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ae204c86-8d93-4adf-988c-cd836e011270-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jf555\" (UID: \"ae204c86-8d93-4adf-988c-cd836e011270\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:46.898332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.897983    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6547c71f-71a2-40c9-912f-9283b6bdb560-serving-cert\") pod \"console-operator-7cd56b8dbc-rhpjv\" (UID: \"6547c71f-71a2-40c9-912f-9283b6bdb560\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:46.898332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898013    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-metrics-tls\") pod \"dns-default-8p8hk\" (UID: \"aadb342f-3310-4c81-967d-0f0a8ef5a4e6\") " pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:46.898332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898042    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-image-registry-private-configuration\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:46.898332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898070    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9046c34b-bb8f-4397-8b9d-7e2095add002-konnectivity-ca\") pod \"konnectivity-agent-cccr5\" (UID: \"9046c34b-bb8f-4397-8b9d-7e2095add002\") " pod="kube-system/konnectivity-agent-cccr5"
Nov 03 21:14:46.898332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898083    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eea6e5c2-57f8-42c6-9ce9-55a3c8f92790-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7\" (UID: \"eea6e5c2-57f8-42c6-9ce9-55a3c8f92790\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7"
Nov 03 21:14:46.898332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898096    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fphb6\" (UniqueName: \"kubernetes.io/projected/5cacf547-c691-4b14-936c-e6b08691bf93-kube-api-access-fphb6\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:46.898332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898145    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6547c71f-71a2-40c9-912f-9283b6bdb560-serving-cert\") pod \"console-operator-7cd56b8dbc-rhpjv\" (UID: \"6547c71f-71a2-40c9-912f-9283b6bdb560\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:46.898332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898139    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a295913f-3811-47d6-9e85-a76182e45649-env-overrides\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.898332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898161    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc77l\" (UniqueName: \"kubernetes.io/projected/cad3df5d-3fe9-449b-8b70-793bceb47985-kube-api-access-wc77l\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:46.898332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898270    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgjr7\" (UniqueName: \"kubernetes.io/projected/52709fc1-b443-4a46-a95e-20cacc4ac4a0-kube-api-access-qgjr7\") pod \"kas-connection-checker-685895c7c9-sds64\" (UID: \"52709fc1-b443-4a46-a95e-20cacc4ac4a0\") " pod="kube-system/kas-connection-checker-685895c7c9-sds64"
Nov 03 21:14:46.898332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898293    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-image-registry-private-configuration\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:46.898332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898301    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ef7a859e-70f5-497a-9ad4-6614885fb71e-image-registry-private-configuration\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:46.898978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898359    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9046c34b-bb8f-4397-8b9d-7e2095add002-konnectivity-ca\") pod \"konnectivity-agent-cccr5\" (UID: \"9046c34b-bb8f-4397-8b9d-7e2095add002\") " pod="kube-system/konnectivity-agent-cccr5"
Nov 03 21:14:46.898978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898394    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:46.898978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898428    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eea6e5c2-57f8-42c6-9ce9-55a3c8f92790-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7\" (UID: \"eea6e5c2-57f8-42c6-9ce9-55a3c8f92790\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7"
Nov 03 21:14:46.898978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898447    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42cwb\" (UniqueName: \"kubernetes.io/projected/bbd3c724-e7e5-4382-b681-adf7fd8f302b-kube-api-access-42cwb\") pod \"kas-connection-checker-685895c7c9-bhpkz\" (UID: \"bbd3c724-e7e5-4382-b681-adf7fd8f302b\") " pod="kube-system/kas-connection-checker-685895c7c9-bhpkz"
Nov 03 21:14:46.898978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898465    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-etc-openvswitch\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.898978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898469    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ef7a859e-70f5-497a-9ad4-6614885fb71e-image-registry-private-configuration\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:46.898978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898480    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23c56cf7-2a99-4df8-841a-d84b6a9a3822-tmp\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:46.898978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898499    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-sys-fs\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:46.898978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898530    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-system-cni-dir\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:46.898978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898555    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-systemd-units\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.898978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898564    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23c56cf7-2a99-4df8-841a-d84b6a9a3822-tmp\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:46.898978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898581    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxb9x\" (UniqueName: \"kubernetes.io/projected/356632db-499f-4ced-bf01-b280472b53f1-kube-api-access-pxb9x\") pod \"migrator-66df5d496b-v6qdv\" (UID: \"356632db-499f-4ced-bf01-b280472b53f1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v6qdv"
Nov 03 21:14:46.898978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898626    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbv62\" (UniqueName: \"kubernetes.io/projected/6547c71f-71a2-40c9-912f-9283b6bdb560-kube-api-access-qbv62\") pod \"console-operator-7cd56b8dbc-rhpjv\" (UID: \"6547c71f-71a2-40c9-912f-9283b6bdb560\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:46.898978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898644    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eea6e5c2-57f8-42c6-9ce9-55a3c8f92790-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7\" (UID: \"eea6e5c2-57f8-42c6-9ce9-55a3c8f92790\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7"
Nov 03 21:14:46.898978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898736    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jf555\" (UID: \"ae204c86-8d93-4adf-988c-cd836e011270\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:46.898978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898755    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-systemd\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:46.898978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898773    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv8v4\" (UniqueName: \"kubernetes.io/projected/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-kube-api-access-pv8v4\") pod \"dns-default-8p8hk\" (UID: \"aadb342f-3310-4c81-967d-0f0a8ef5a4e6\") " pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:46.898978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898804    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-trusted-ca\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:46.898978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898947    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-var-lib-cni-multus\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:46.898978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898976    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-registration-dir\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:46.899705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.898991    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cad3df5d-3fe9-449b-8b70-793bceb47985-crio-socket\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:46.899705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899014    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-stats-auth\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:46.899705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899041    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j2ksl\" (UID: \"bb8a800c-36be-4da0-b1e6-5ef65f994fc1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j2ksl"
Nov 03 21:14:46.899705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899082    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42d4m\" (UniqueName: \"kubernetes.io/projected/a295913f-3811-47d6-9e85-a76182e45649-kube-api-access-42d4m\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.899705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899110    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5cacf547-c691-4b14-936c-e6b08691bf93-serving-cert\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:46.899705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899133    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-kubernetes\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:46.899705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899174    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/abf1a25c-1da1-449d-90c0-35d6fc406647-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:46.899705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899200    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:46.899705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899298    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5cacf547-c691-4b14-936c-e6b08691bf93-serving-cert\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:46.899705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899298    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-stats-auth\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:46.899705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899325    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-kubelet-config\") pod \"global-pull-secret-syncer-v5vrf\" (UID: \"96fee653-58b4-46bf-9ed5-0d3c5a93e8a8\") " pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:46.899705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899371    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6547c71f-71a2-40c9-912f-9283b6bdb560-config\") pod \"console-operator-7cd56b8dbc-rhpjv\" (UID: \"6547c71f-71a2-40c9-912f-9283b6bdb560\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:46.899705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899420    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs262\" (UniqueName: \"kubernetes.io/projected/7ccca940-de12-451b-b920-d6bef516942f-kube-api-access-fs262\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:46.899705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899439    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/abf1a25c-1da1-449d-90c0-35d6fc406647-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:46.899705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899488    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f63c6326-4f32-4fa3-925a-83743ef7ff39-cni-binary-copy\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:46.899705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899542    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e703c9a2-a51a-4de6-b30c-6cc2aa810e57-host-slash\") pod \"iptables-alerter-km5mx\" (UID: \"e703c9a2-a51a-4de6-b30c-6cc2aa810e57\") " pod="openshift-network-operator/iptables-alerter-km5mx"
Nov 03 21:14:46.899705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899569    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-log-socket\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.899705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899654    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-cni-netd\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.899705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899702    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j2ksl\" (UID: \"bb8a800c-36be-4da0-b1e6-5ef65f994fc1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j2ksl"
Nov 03 21:14:46.900417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899705    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6fs8\" (UniqueName: \"kubernetes.io/projected/c3560fa9-2666-4e0d-854c-3c126489c61d-kube-api-access-l6fs8\") pod \"node-ca-z5s4n\" (UID: \"c3560fa9-2666-4e0d-854c-3c126489c61d\") " pod="openshift-image-registry/node-ca-z5s4n"
Nov 03 21:14:46.900417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899770    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzr7h\" (UniqueName: \"kubernetes.io/projected/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-kube-api-access-fzr7h\") pod \"cluster-samples-operator-6ddb6c8b94-sqlqz\" (UID: \"cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"
Nov 03 21:14:46.900417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899801    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-sys\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:46.900417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899851    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f63c6326-4f32-4fa3-925a-83743ef7ff39-cni-binary-copy\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:46.900417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899944    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4hxv\" (UniqueName: \"kubernetes.io/projected/440dc059-df07-4f3e-a595-50d35d172e8c-kube-api-access-f4hxv\") pod \"service-ca-5fffc97bdf-zf7tn\" (UID: \"440dc059-df07-4f3e-a595-50d35d172e8c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zf7tn"
Nov 03 21:14:46.900417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899971    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-socket-dir\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:46.900417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.899970    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6547c71f-71a2-40c9-912f-9283b6bdb560-config\") pod \"console-operator-7cd56b8dbc-rhpjv\" (UID: \"6547c71f-71a2-40c9-912f-9283b6bdb560\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:46.900417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900001    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa-serving-cert\") pod \"service-ca-operator-544d749c8c-k65s8\" (UID: \"067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8"
Nov 03 21:14:46.900417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900022    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e703c9a2-a51a-4de6-b30c-6cc2aa810e57-iptables-alerter-script\") pod \"iptables-alerter-km5mx\" (UID: \"e703c9a2-a51a-4de6-b30c-6cc2aa810e57\") " pod="openshift-network-operator/iptables-alerter-km5mx"
Nov 03 21:14:46.900417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900047    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-kubelet\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.900417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900074    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5cacf547-c691-4b14-936c-e6b08691bf93-snapshots\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:46.900417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900112    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-certificates\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:46.900417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900158    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mddbv\" (UniqueName: \"kubernetes.io/projected/caaa9f42-f8a2-4459-8917-a53e8fd852cc-kube-api-access-mddbv\") pod \"periodic-gathering-g82mg-mqwqf\" (UID: \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\") " pod="openshift-insights/periodic-gathering-g82mg-mqwqf"
Nov 03 21:14:46.900417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900168    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5cacf547-c691-4b14-936c-e6b08691bf93-snapshots\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:46.900417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900176    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa-serving-cert\") pod \"service-ca-operator-544d749c8c-k65s8\" (UID: \"067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8"
Nov 03 21:14:46.900417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900207    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ef7a859e-70f5-497a-9ad4-6614885fb71e-installation-pull-secrets\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:46.900417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900260    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhskl\" (UniqueName: \"kubernetes.io/projected/e703c9a2-a51a-4de6-b30c-6cc2aa810e57-kube-api-access-dhskl\") pod \"iptables-alerter-km5mx\" (UID: \"e703c9a2-a51a-4de6-b30c-6cc2aa810e57\") " pod="openshift-network-operator/iptables-alerter-km5mx"
Nov 03 21:14:46.900417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900288    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt2rs\" (UniqueName: \"kubernetes.io/projected/23c56cf7-2a99-4df8-841a-d84b6a9a3822-kube-api-access-kt2rs\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:46.900417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900311    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e703c9a2-a51a-4de6-b30c-6cc2aa810e57-iptables-alerter-script\") pod \"iptables-alerter-km5mx\" (UID: \"e703c9a2-a51a-4de6-b30c-6cc2aa810e57\") " pod="openshift-network-operator/iptables-alerter-km5mx"
Nov 03 21:14:46.900417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900313    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:46.900417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900354    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cad3df5d-3fe9-449b-8b70-793bceb47985-data-volume\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:46.900417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900399    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ef7a859e-70f5-497a-9ad4-6614885fb71e-installation-pull-secrets\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:46.901215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900511    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p2xx\" (UniqueName: \"kubernetes.io/projected/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-kube-api-access-2p2xx\") pod \"network-metrics-daemon-6xvgt\" (UID: \"4b84ba29-a116-48ef-8d9f-d81ceb4b3359\") " pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:46.901215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900543    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cad3df5d-3fe9-449b-8b70-793bceb47985-data-volume\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:46.901215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900578    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abf1a25c-1da1-449d-90c0-35d6fc406647-cnibin\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:46.901215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900599    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv52h\" (UniqueName: \"kubernetes.io/projected/abf1a25c-1da1-449d-90c0-35d6fc406647-kube-api-access-tv52h\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:46.901215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900638    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-run-openvswitch\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.901215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900670    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d34868b-605a-4bfb-81a3-432e0db0ce6a-tmp-dir\") pod \"node-resolver-554sk\" (UID: \"3d34868b-605a-4bfb-81a3-432e0db0ce6a\") " pod="openshift-dns/node-resolver-554sk"
Nov 03 21:14:46.901215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900694    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-var-lib-kubelet\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:46.901215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900734    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3d34868b-605a-4bfb-81a3-432e0db0ce6a-hosts-file\") pod \"node-resolver-554sk\" (UID: \"3d34868b-605a-4bfb-81a3-432e0db0ce6a\") " pod="openshift-dns/node-resolver-554sk"
Nov 03 21:14:46.901215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900766    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caaa9f42-f8a2-4459-8917-a53e8fd852cc-service-ca-bundle\") pod \"periodic-gathering-g82mg-mqwqf\" (UID: \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\") " pod="openshift-insights/periodic-gathering-g82mg-mqwqf"
Nov 03 21:14:46.901215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900772    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d34868b-605a-4bfb-81a3-432e0db0ce6a-tmp-dir\") pod \"node-resolver-554sk\" (UID: \"3d34868b-605a-4bfb-81a3-432e0db0ce6a\") " pod="openshift-dns/node-resolver-554sk"
Nov 03 21:14:46.901215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900790    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-dbus\") pod \"global-pull-secret-syncer-v5vrf\" (UID: \"96fee653-58b4-46bf-9ed5-0d3c5a93e8a8\") " pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:46.901215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900818    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-run-netns\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:46.901215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900849    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-slash\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.901215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900853    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-certificates\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:46.901215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900875    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:46.901215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900897    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-node-log\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.901215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900897    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caaa9f42-f8a2-4459-8917-a53e8fd852cc-service-ca-bundle\") pod \"periodic-gathering-g82mg-mqwqf\" (UID: \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\") " pod="openshift-insights/periodic-gathering-g82mg-mqwqf"
Nov 03 21:14:46.901215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900916    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcxtm\" (UniqueName: \"kubernetes.io/projected/ae204c86-8d93-4adf-988c-cd836e011270-kube-api-access-hcxtm\") pod \"cluster-monitoring-operator-884576c57-jf555\" (UID: \"ae204c86-8d93-4adf-988c-cd836e011270\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:46.901215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900954    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-sysconfig\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:46.901215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.900992    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-host\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:46.901215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.901101    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abf1a25c-1da1-449d-90c0-35d6fc406647-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:46.901215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.901133    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c3560fa9-2666-4e0d-854c-3c126489c61d-serviceca\") pod \"node-ca-z5s4n\" (UID: \"c3560fa9-2666-4e0d-854c-3c126489c61d\") " pod="openshift-image-registry/node-ca-z5s4n"
Nov 03 21:14:46.901215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.901150    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-run-k8s-cni-cncf-io\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:46.902161 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.901300    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-var-lib-openvswitch\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.902161 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.901342    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-cni-bin\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.902161 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.901370    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-ca-trust-extracted\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:46.902161 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.901393    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f63c6326-4f32-4fa3-925a-83743ef7ff39-multus-daemon-config\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:46.902161 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.901413    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7qxh\" (UniqueName: \"kubernetes.io/projected/f63c6326-4f32-4fa3-925a-83743ef7ff39-kube-api-access-q7qxh\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:46.902161 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.901439    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sqlqz\" (UID: \"cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"
Nov 03 21:14:46.902161 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.901490    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-run-ovn-kubernetes\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.902161 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.901504    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-ca-trust-extracted\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:46.902161 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.901595    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a295913f-3811-47d6-9e85-a76182e45649-ovnkube-script-lib\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.902161 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.901947    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f63c6326-4f32-4fa3-925a-83743ef7ff39-multus-daemon-config\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:46.902621 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.902167    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c3560fa9-2666-4e0d-854c-3c126489c61d-serviceca\") pod \"node-ca-z5s4n\" (UID: \"c3560fa9-2666-4e0d-854c-3c126489c61d\") " pod="openshift-image-registry/node-ca-z5s4n"
Nov 03 21:14:46.902621 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.902340    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a295913f-3811-47d6-9e85-a76182e45649-ovnkube-script-lib\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.902740 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.902719    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c27ff312-c8d3-4691-b535-ea8adf44d9c3-cert\") pod \"ingress-canary-m5s9b\" (UID: \"c27ff312-c8d3-4691-b535-ea8adf44d9c3\") " pod="openshift-ingress-canary/ingress-canary-m5s9b"
Nov 03 21:14:46.902788 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.902763    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-lib-modules\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:46.902834 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.902789    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cad3df5d-3fe9-449b-8b70-793bceb47985-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:46.902834 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.902826    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp68p\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-kube-api-access-lp68p\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:46.902928 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.902857    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhdwv\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-kube-api-access-bhdwv\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:46.902928 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.902888    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-default-certificate\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:46.902928 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.902917    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-run-netns\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.903055 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.902941    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-modprobe-d\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:46.903055 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.902979    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-sysctl-d\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:46.903055 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.903004    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abf1a25c-1da1-449d-90c0-35d6fc406647-cni-binary-copy\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:46.903055 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.903034    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef7a859e-70f5-497a-9ad4-6614885fb71e-trusted-ca\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:46.903222 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.903063    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-cnibin\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:46.903222 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.903095    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpzsk\" (UniqueName: \"kubernetes.io/projected/c27ff312-c8d3-4691-b535-ea8adf44d9c3-kube-api-access-zpzsk\") pod \"ingress-canary-m5s9b\" (UID: \"c27ff312-c8d3-4691-b535-ea8adf44d9c3\") " pod="openshift-ingress-canary/ingress-canary-m5s9b"
Nov 03 21:14:46.903222 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.903118    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caaa9f42-f8a2-4459-8917-a53e8fd852cc-trusted-ca-bundle\") pod \"periodic-gathering-g82mg-mqwqf\" (UID: \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\") " pod="openshift-insights/periodic-gathering-g82mg-mqwqf"
Nov 03 21:14:46.903222 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.903149    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjp6p\" (UniqueName: \"kubernetes.io/projected/067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa-kube-api-access-fjp6p\") pod \"service-ca-operator-544d749c8c-k65s8\" (UID: \"067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8"
Nov 03 21:14:46.903222 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.903177    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxp6s\" (UniqueName: \"kubernetes.io/projected/3d34868b-605a-4bfb-81a3-432e0db0ce6a-kube-api-access-wxp6s\") pod \"node-resolver-554sk\" (UID: \"3d34868b-605a-4bfb-81a3-432e0db0ce6a\") " pod="openshift-dns/node-resolver-554sk"
Nov 03 21:14:46.903222 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.903206    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/caaa9f42-f8a2-4459-8917-a53e8fd852cc-archives-path\") pod \"periodic-gathering-g82mg-mqwqf\" (UID: \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\") " pod="openshift-insights/periodic-gathering-g82mg-mqwqf"
Nov 03 21:14:46.903505 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.903252    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-sysctl-conf\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:46.903505 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.903293    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-config-volume\") pod \"dns-default-8p8hk\" (UID: \"aadb342f-3310-4c81-967d-0f0a8ef5a4e6\") " pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:46.903505 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.903325    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jrqs\" (UniqueName: \"kubernetes.io/projected/06a672d7-0667-4bab-98ee-91dd71dfc489-kube-api-access-9jrqs\") pod \"network-check-source-6f78d59f99-sb5xl\" (UID: \"06a672d7-0667-4bab-98ee-91dd71dfc489\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sb5xl"
Nov 03 21:14:46.903505 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.903359    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:46.903505 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.903390    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa-config\") pod \"service-ca-operator-544d749c8c-k65s8\" (UID: \"067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8"
Nov 03 21:14:46.903505 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.903422    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-run-systemd\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.903505 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.903449    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zn62\" (UniqueName: \"kubernetes.io/projected/974ed564-6386-4e1e-9326-f14ccc36658c-kube-api-access-4zn62\") pod \"kas-connection-checker-685895c7c9-27lr2\" (UID: \"974ed564-6386-4e1e-9326-f14ccc36658c\") " pod="kube-system/kas-connection-checker-685895c7c9-27lr2"
Nov 03 21:14:46.903505 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.903483    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cacf547-c691-4b14-936c-e6b08691bf93-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:46.903844 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.903595    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 21:14:46.903844 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.903689    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-tuned\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:46.903844 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.903785    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-tuned\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:46.904211 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.904193    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c27ff312-c8d3-4691-b535-ea8adf44d9c3-cert\") pod \"ingress-canary-m5s9b\" (UID: \"c27ff312-c8d3-4691-b535-ea8adf44d9c3\") " pod="openshift-ingress-canary/ingress-canary-m5s9b"
Nov 03 21:14:46.904389 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.904368    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/caaa9f42-f8a2-4459-8917-a53e8fd852cc-archives-path\") pod \"periodic-gathering-g82mg-mqwqf\" (UID: \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\") " pod="openshift-insights/periodic-gathering-g82mg-mqwqf"
Nov 03 21:14:46.904546 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.904528    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-default-certificate\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:46.904597 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.904562    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 21:14:46.904831 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.904813    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa-config\") pod \"service-ca-operator-544d749c8c-k65s8\" (UID: \"067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8"
Nov 03 21:14:46.904982 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.904962    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef7a859e-70f5-497a-9ad4-6614885fb71e-trusted-ca\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:46.905043 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.905027    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abf1a25c-1da1-449d-90c0-35d6fc406647-cni-binary-copy\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:46.905088 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.905044    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-tmp-dir\") pod \"dns-default-8p8hk\" (UID: \"aadb342f-3310-4c81-967d-0f0a8ef5a4e6\") " pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:46.905088 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.905075    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-bound-sa-token\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:46.905172 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.905108    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-sqlqz\" (UID: \"cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"
Nov 03 21:14:46.905172 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.905140    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-run-ovn\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.905347 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.905170    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a295913f-3811-47d6-9e85-a76182e45649-ovn-node-metrics-cert\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.905347 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.905197    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cacf547-c691-4b14-936c-e6b08691bf93-tmp\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:46.905347 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.905227    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-hostroot\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:46.905347 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.905271    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-config-volume\") pod \"dns-default-8p8hk\" (UID: \"aadb342f-3310-4c81-967d-0f0a8ef5a4e6\") " pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:46.905347 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.905289    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a295913f-3811-47d6-9e85-a76182e45649-ovnkube-config\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.905561 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.905494    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a295913f-3811-47d6-9e85-a76182e45649-ovnkube-config\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.905561 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.905494    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-tmp-dir\") pod \"dns-default-8p8hk\" (UID: \"aadb342f-3310-4c81-967d-0f0a8ef5a4e6\") " pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:46.905946 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.905924    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a295913f-3811-47d6-9e85-a76182e45649-ovn-node-metrics-cert\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.906216 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.906199    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cacf547-c691-4b14-936c-e6b08691bf93-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:46.906602 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.906584    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-sqlqz\" (UID: \"cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"
Nov 03 21:14:46.907157 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.907036    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cacf547-c691-4b14-936c-e6b08691bf93-tmp\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:46.909516 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.909498    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-trusted-ca\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:46.914189 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.914172    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caaa9f42-f8a2-4459-8917-a53e8fd852cc-trusted-ca-bundle\") pod \"periodic-gathering-g82mg-mqwqf\" (UID: \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\") " pod="openshift-insights/periodic-gathering-g82mg-mqwqf"
Nov 03 21:14:46.952424 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.952387    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jchm\" (UniqueName: \"kubernetes.io/projected/2000161a-2aa8-4638-a4f7-fbe305e929ae-kube-api-access-9jchm\") pod \"migrator-66df5d496b-jqkbh\" (UID: \"2000161a-2aa8-4638-a4f7-fbe305e929ae\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jqkbh"
Nov 03 21:14:46.952567 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.952552    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb9kd\" (UniqueName: \"kubernetes.io/projected/784231bb-189f-472c-83ed-6ced092f10a1-kube-api-access-nb9kd\") pod \"network-check-target-8w5st\" (UID: \"784231bb-189f-472c-83ed-6ced092f10a1\") " pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:46.952652 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.952627    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-bound-sa-token\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:46.952766 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.952749    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvsxw\" (UniqueName: \"kubernetes.io/projected/35650ff5-011e-4572-886f-8bf47f598219-kube-api-access-gvsxw\") pod \"volume-data-source-validator-58c9f5f46d-nv9qq\" (UID: \"35650ff5-011e-4572-886f-8bf47f598219\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nv9qq"
Nov 03 21:14:46.952800 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.952765    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4vrx\" (UniqueName: \"kubernetes.io/projected/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-kube-api-access-m4vrx\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:46.956129 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.956106    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m22kb\" (UniqueName: \"kubernetes.io/projected/eea6e5c2-57f8-42c6-9ce9-55a3c8f92790-kube-api-access-m22kb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7\" (UID: \"eea6e5c2-57f8-42c6-9ce9-55a3c8f92790\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7"
Nov 03 21:14:46.956385 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.956355    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgjr7\" (UniqueName: \"kubernetes.io/projected/52709fc1-b443-4a46-a95e-20cacc4ac4a0-kube-api-access-qgjr7\") pod \"kas-connection-checker-685895c7c9-sds64\" (UID: \"52709fc1-b443-4a46-a95e-20cacc4ac4a0\") " pod="kube-system/kas-connection-checker-685895c7c9-sds64"
Nov 03 21:14:46.956989 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.956547    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42d4m\" (UniqueName: \"kubernetes.io/projected/a295913f-3811-47d6-9e85-a76182e45649-kube-api-access-42d4m\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:46.956989 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.956710    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxb9x\" (UniqueName: \"kubernetes.io/projected/356632db-499f-4ced-bf01-b280472b53f1-kube-api-access-pxb9x\") pod \"migrator-66df5d496b-v6qdv\" (UID: \"356632db-499f-4ced-bf01-b280472b53f1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v6qdv"
Nov 03 21:14:46.956989 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.956751    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzr7h\" (UniqueName: \"kubernetes.io/projected/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-kube-api-access-fzr7h\") pod \"cluster-samples-operator-6ddb6c8b94-sqlqz\" (UID: \"cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"
Nov 03 21:14:46.956989 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.956765    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbv62\" (UniqueName: \"kubernetes.io/projected/6547c71f-71a2-40c9-912f-9283b6bdb560-kube-api-access-qbv62\") pod \"console-operator-7cd56b8dbc-rhpjv\" (UID: \"6547c71f-71a2-40c9-912f-9283b6bdb560\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:46.956989 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.956936    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv8v4\" (UniqueName: \"kubernetes.io/projected/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-kube-api-access-pv8v4\") pod \"dns-default-8p8hk\" (UID: \"aadb342f-3310-4c81-967d-0f0a8ef5a4e6\") " pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:46.957305 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.957040    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc77l\" (UniqueName: \"kubernetes.io/projected/cad3df5d-3fe9-449b-8b70-793bceb47985-kube-api-access-wc77l\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:46.957305 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.957051    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs262\" (UniqueName: \"kubernetes.io/projected/7ccca940-de12-451b-b920-d6bef516942f-kube-api-access-fs262\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:46.957305 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.957140    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mddbv\" (UniqueName: \"kubernetes.io/projected/caaa9f42-f8a2-4459-8917-a53e8fd852cc-kube-api-access-mddbv\") pod \"periodic-gathering-g82mg-mqwqf\" (UID: \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\") " pod="openshift-insights/periodic-gathering-g82mg-mqwqf"
Nov 03 21:14:46.957305 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.957168    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4hxv\" (UniqueName: \"kubernetes.io/projected/440dc059-df07-4f3e-a595-50d35d172e8c-kube-api-access-f4hxv\") pod \"service-ca-5fffc97bdf-zf7tn\" (UID: \"440dc059-df07-4f3e-a595-50d35d172e8c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zf7tn"
Nov 03 21:14:46.957305 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.957292    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fphb6\" (UniqueName: \"kubernetes.io/projected/5cacf547-c691-4b14-936c-e6b08691bf93-kube-api-access-fphb6\") pod \"insights-operator-7c4986d449-bc6rw\" (UID: \"5cacf547-c691-4b14-936c-e6b08691bf93\") " pod="openshift-insights/insights-operator-7c4986d449-bc6rw"
Nov 03 21:14:46.958021 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.957999    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6fs8\" (UniqueName: \"kubernetes.io/projected/c3560fa9-2666-4e0d-854c-3c126489c61d-kube-api-access-l6fs8\") pod \"node-ca-z5s4n\" (UID: \"c3560fa9-2666-4e0d-854c-3c126489c61d\") " pod="openshift-image-registry/node-ca-z5s4n"
Nov 03 21:14:46.958460 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.958429    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv52h\" (UniqueName: \"kubernetes.io/projected/abf1a25c-1da1-449d-90c0-35d6fc406647-kube-api-access-tv52h\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:46.958552 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.958532    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7qxh\" (UniqueName: \"kubernetes.io/projected/f63c6326-4f32-4fa3-925a-83743ef7ff39-kube-api-access-q7qxh\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:46.958552 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.958546    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42cwb\" (UniqueName: \"kubernetes.io/projected/bbd3c724-e7e5-4382-b681-adf7fd8f302b-kube-api-access-42cwb\") pod \"kas-connection-checker-685895c7c9-bhpkz\" (UID: \"bbd3c724-e7e5-4382-b681-adf7fd8f302b\") " pod="kube-system/kas-connection-checker-685895c7c9-bhpkz"
Nov 03 21:14:46.958754 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.958733    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhskl\" (UniqueName: \"kubernetes.io/projected/e703c9a2-a51a-4de6-b30c-6cc2aa810e57-kube-api-access-dhskl\") pod \"iptables-alerter-km5mx\" (UID: \"e703c9a2-a51a-4de6-b30c-6cc2aa810e57\") " pod="openshift-network-operator/iptables-alerter-km5mx"
Nov 03 21:14:46.958898 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.958876    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p2xx\" (UniqueName: \"kubernetes.io/projected/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-kube-api-access-2p2xx\") pod \"network-metrics-daemon-6xvgt\" (UID: \"4b84ba29-a116-48ef-8d9f-d81ceb4b3359\") " pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:46.958968 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.958912    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt2rs\" (UniqueName: \"kubernetes.io/projected/23c56cf7-2a99-4df8-841a-d84b6a9a3822-kube-api-access-kt2rs\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:46.959027 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.958966    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcxtm\" (UniqueName: \"kubernetes.io/projected/ae204c86-8d93-4adf-988c-cd836e011270-kube-api-access-hcxtm\") pod \"cluster-monitoring-operator-884576c57-jf555\" (UID: \"ae204c86-8d93-4adf-988c-cd836e011270\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:46.959423 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.959404    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjp6p\" (UniqueName: \"kubernetes.io/projected/067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa-kube-api-access-fjp6p\") pod \"service-ca-operator-544d749c8c-k65s8\" (UID: \"067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8"
Nov 03 21:14:46.959577 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.959562    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpzsk\" (UniqueName: \"kubernetes.io/projected/c27ff312-c8d3-4691-b535-ea8adf44d9c3-kube-api-access-zpzsk\") pod \"ingress-canary-m5s9b\" (UID: \"c27ff312-c8d3-4691-b535-ea8adf44d9c3\") " pod="openshift-ingress-canary/ingress-canary-m5s9b"
Nov 03 21:14:46.959739 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.959723    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxp6s\" (UniqueName: \"kubernetes.io/projected/3d34868b-605a-4bfb-81a3-432e0db0ce6a-kube-api-access-wxp6s\") pod \"node-resolver-554sk\" (UID: \"3d34868b-605a-4bfb-81a3-432e0db0ce6a\") " pod="openshift-dns/node-resolver-554sk"
Nov 03 21:14:46.960662 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.960644    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-bound-sa-token\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:46.960764 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.960738    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zn62\" (UniqueName: \"kubernetes.io/projected/974ed564-6386-4e1e-9326-f14ccc36658c-kube-api-access-4zn62\") pod \"kas-connection-checker-685895c7c9-27lr2\" (UID: \"974ed564-6386-4e1e-9326-f14ccc36658c\") " pod="kube-system/kas-connection-checker-685895c7c9-27lr2"
Nov 03 21:14:46.960764 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.960758    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhdwv\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-kube-api-access-bhdwv\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:46.961029 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.961011    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp68p\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-kube-api-access-lp68p\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:46.961310 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:46.961296    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jrqs\" (UniqueName: \"kubernetes.io/projected/06a672d7-0667-4bab-98ee-91dd71dfc489-kube-api-access-9jrqs\") pod \"network-check-source-6f78d59f99-sb5xl\" (UID: \"06a672d7-0667-4bab-98ee-91dd71dfc489\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sb5xl"
Nov 03 21:14:47.006257 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006210    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:47.006257 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006262    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-etc-openvswitch\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.006493 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006280    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-sys-fs\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:47.006493 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006298    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-system-cni-dir\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.006493 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006321    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-systemd-units\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.006493 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006348    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jf555\" (UID: \"ae204c86-8d93-4adf-988c-cd836e011270\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:47.006493 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006373    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-systemd\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:47.006493 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006376    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-sys-fs\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:47.006493 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006385    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-systemd-units\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.006493 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.006393    5053 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 21:14:47.006493 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.006415    5053 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75cb56744b-c6rsx: secret "image-registry-tls" not found
Nov 03 21:14:47.006493 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006398    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-var-lib-cni-multus\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.006493 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006443    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-systemd\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:47.006493 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.006462    5053 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 21:14:47.006493 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006346    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-etc-openvswitch\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.006493 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006378    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-system-cni-dir\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.006493 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006469    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-var-lib-cni-multus\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.006493 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006461    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-registration-dir\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:47.006493 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.006499    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls podName:ef7a859e-70f5-497a-9ad4-6614885fb71e nodeName:}" failed. No retries permitted until 2026-11-03 21:14:47.506477171 +0000 UTC m=+2.000399043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls") pod "image-registry-75cb56744b-c6rsx" (UID: "ef7a859e-70f5-497a-9ad4-6614885fb71e") : secret "image-registry-tls" not found
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006526    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-registration-dir\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.006565    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls podName:ae204c86-8d93-4adf-988c-cd836e011270 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:47.506550651 +0000 UTC m=+2.000472505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jf555" (UID: "ae204c86-8d93-4adf-988c-cd836e011270") : secret "cluster-monitoring-operator-tls" not found
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006579    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cad3df5d-3fe9-449b-8b70-793bceb47985-crio-socket\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006602    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-kubernetes\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006626    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006641    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cad3df5d-3fe9-449b-8b70-793bceb47985-crio-socket\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006644    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-kubelet-config\") pod \"global-pull-secret-syncer-v5vrf\" (UID: \"96fee653-58b4-46bf-9ed5-0d3c5a93e8a8\") " pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006672    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-kubelet-config\") pod \"global-pull-secret-syncer-v5vrf\" (UID: \"96fee653-58b4-46bf-9ed5-0d3c5a93e8a8\") " pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006682    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e703c9a2-a51a-4de6-b30c-6cc2aa810e57-host-slash\") pod \"iptables-alerter-km5mx\" (UID: \"e703c9a2-a51a-4de6-b30c-6cc2aa810e57\") " pod="openshift-network-operator/iptables-alerter-km5mx"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006705    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-kubernetes\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006704    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-log-socket\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006718    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006728    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e703c9a2-a51a-4de6-b30c-6cc2aa810e57-host-slash\") pod \"iptables-alerter-km5mx\" (UID: \"e703c9a2-a51a-4de6-b30c-6cc2aa810e57\") " pod="openshift-network-operator/iptables-alerter-km5mx"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006734    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-log-socket\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006737    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-cni-netd\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006767    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-cni-netd\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006796    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-sys\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006819    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-socket-dir\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006847    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-kubelet\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006848    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-sys\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006871    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006891    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abf1a25c-1da1-449d-90c0-35d6fc406647-cnibin\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006907    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-run-openvswitch\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006919    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-kubelet\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006940    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006939    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-socket-dir\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006944    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abf1a25c-1da1-449d-90c0-35d6fc406647-cnibin\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006947    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-run-openvswitch\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006962    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-var-lib-kubelet\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.006986    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-var-lib-kubelet\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007007    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3d34868b-605a-4bfb-81a3-432e0db0ce6a-hosts-file\") pod \"node-resolver-554sk\" (UID: \"3d34868b-605a-4bfb-81a3-432e0db0ce6a\") " pod="openshift-dns/node-resolver-554sk"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007033    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-dbus\") pod \"global-pull-secret-syncer-v5vrf\" (UID: \"96fee653-58b4-46bf-9ed5-0d3c5a93e8a8\") " pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007056    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-run-netns\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007084    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3d34868b-605a-4bfb-81a3-432e0db0ce6a-hosts-file\") pod \"node-resolver-554sk\" (UID: \"3d34868b-605a-4bfb-81a3-432e0db0ce6a\") " pod="openshift-dns/node-resolver-554sk"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007094    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96fee653-58b4-46bf-9ed5-0d3c5a93e8a8-dbus\") pod \"global-pull-secret-syncer-v5vrf\" (UID: \"96fee653-58b4-46bf-9ed5-0d3c5a93e8a8\") " pod="kube-system/global-pull-secret-syncer-v5vrf"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007084    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-slash\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007119    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007135    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-node-log\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007149    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-sysconfig\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007150    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-run-netns\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007162    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-host\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007177    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abf1a25c-1da1-449d-90c0-35d6fc406647-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007191    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-run-k8s-cni-cncf-io\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007206    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-var-lib-openvswitch\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007227    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-host\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007230    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-sysconfig\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007252    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-cni-bin\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.007224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007286    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-var-lib-openvswitch\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007190    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-slash\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007298    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sqlqz\" (UID: \"cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007298    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abf1a25c-1da1-449d-90c0-35d6fc406647-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007323    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-cni-bin\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007325    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-run-ovn-kubernetes\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007357    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-lib-modules\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007382    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cad3df5d-3fe9-449b-8b70-793bceb47985-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007388    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-node-log\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007406    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-run-netns\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007425    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-run-k8s-cni-cncf-io\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007359    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-run-ovn-kubernetes\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.007305    5053 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007425    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-modprobe-d\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007468    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-lib-modules\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007481    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-sysctl-d\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007474    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-run-netns\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.007504    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls podName:cad3df5d-3fe9-449b-8b70-793bceb47985 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:47.507489124 +0000 UTC m=+2.001410996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9vxl" (UID: "cad3df5d-3fe9-449b-8b70-793bceb47985") : secret "insights-runtime-extractor-tls" not found
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007548    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-cnibin\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007555    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-modprobe-d\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007564    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-sysctl-d\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007585    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-sysctl-conf\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007603    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-cnibin\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007606    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007646    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-run-systemd\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.007682    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle podName:7ccca940-de12-451b-b920-d6bef516942f nodeName:}" failed. No retries permitted until 2026-11-03 21:14:47.507669215 +0000 UTC m=+2.001591085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle") pod "router-default-5ffbbbfcf7-99qbj" (UID: "7ccca940-de12-451b-b920-d6bef516942f") : configmap references non-existent config key: service-ca.crt
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007665    5053 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007697    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-run-systemd\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007701    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-run-ovn\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007729    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-etc-sysctl-conf\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007729    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-hostroot\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007751    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-run-ovn\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007759    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abf1a25c-1da1-449d-90c0-35d6fc406647-system-cni-dir\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007758    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-hostroot\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007792    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c3560fa9-2666-4e0d-854c-3c126489c61d-host\") pod \"node-ca-z5s4n\" (UID: \"c3560fa9-2666-4e0d-854c-3c126489c61d\") " pod="openshift-image-registry/node-ca-z5s4n"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007808    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abf1a25c-1da1-449d-90c0-35d6fc406647-system-cni-dir\") pod \"multus-additional-cni-plugins-f2pgd\" (UID: \"abf1a25c-1da1-449d-90c0-35d6fc406647\") " pod="openshift-multus/multus-additional-cni-plugins-f2pgd"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007820    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-run-multus-certs\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007846    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-var-lib-kubelet\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007866    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c3560fa9-2666-4e0d-854c-3c126489c61d-host\") pod \"node-ca-z5s4n\" (UID: \"c3560fa9-2666-4e0d-854c-3c126489c61d\") " pod="openshift-image-registry/node-ca-z5s4n"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007884    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-var-lib-kubelet\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007903    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-run-multus-certs\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007932    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-multus-conf-dir\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007956    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-device-dir\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007982    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-var-lib-cni-bin\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.007994    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cad3df5d-3fe9-449b-8b70-793bceb47985-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.008017    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-host-var-lib-cni-bin\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.008026    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f83a41bd-5f2e-4c82-a357-ddb5be4be2f6-device-dir\") pod \"aws-ebs-csi-driver-node-thzqr\" (UID: \"f83a41bd-5f2e-4c82-a357-ddb5be4be2f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thzqr"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.008025    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-multus-conf-dir\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.008033    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.008073    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-multus-socket-dir-parent\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.008088    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.008104    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-run\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.008121    5053 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.008123    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs\") pod \"network-metrics-daemon-6xvgt\" (UID: \"4b84ba29-a116-48ef-8d9f-d81ceb4b3359\") " pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.008142    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-etc-kubernetes\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.008141    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-multus-socket-dir-parent\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.008141    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a295913f-3811-47d6-9e85-a76182e45649-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fvkn6\" (UID: \"a295913f-3811-47d6-9e85-a76182e45649\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.008162    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-etc-kubernetes\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.008162    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs podName:7ccca940-de12-451b-b920-d6bef516942f nodeName:}" failed. No retries permitted until 2026-11-03 21:14:47.508148796 +0000 UTC m=+2.002070652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs") pod "router-default-5ffbbbfcf7-99qbj" (UID: "7ccca940-de12-451b-b920-d6bef516942f") : secret "router-metrics-certs-default" not found
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.008221    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.008267    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-multus-cni-dir\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.008293    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j2ksl\" (UID: \"bb8a800c-36be-4da0-b1e6-5ef65f994fc1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j2ksl"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.008320    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-metrics-tls\") pod \"dns-default-8p8hk\" (UID: \"aadb342f-3310-4c81-967d-0f0a8ef5a4e6\") " pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.008353    5053 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.008358    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f63c6326-4f32-4fa3-925a-83743ef7ff39-multus-cni-dir\") pod \"multus-67blc\" (UID: \"f63c6326-4f32-4fa3-925a-83743ef7ff39\") " pod="openshift-multus/multus-67blc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.008367    5053 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6df897dfcf-2b92c: secret "image-registry-tls" not found
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.008358    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/23c56cf7-2a99-4df8-841a-d84b6a9a3822-run\") pod \"tuned-6zndc\" (UID: \"23c56cf7-2a99-4df8-841a-d84b6a9a3822\") " pod="openshift-cluster-node-tuning-operator/tuned-6zndc"
Nov 03 21:14:47.009612 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.008408    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls podName:1db07cc2-aa71-490e-a21a-0a8cb06d94d9 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:47.508396601 +0000 UTC m=+2.002318457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls") pod "image-registry-6df897dfcf-2b92c" (UID: "1db07cc2-aa71-490e-a21a-0a8cb06d94d9") : secret "image-registry-tls" not found
Nov 03 21:14:47.011968 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.010430    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sqlqz\" (UID: \"cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"
Nov 03 21:14:47.011968 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.010763    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb8a800c-36be-4da0-b1e6-5ef65f994fc1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j2ksl\" (UID: \"bb8a800c-36be-4da0-b1e6-5ef65f994fc1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j2ksl"
Nov 03 21:14:47.011968 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.010791    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aadb342f-3310-4c81-967d-0f0a8ef5a4e6-metrics-tls\") pod \"dns-default-8p8hk\" (UID: \"aadb342f-3310-4c81-967d-0f0a8ef5a4e6\") " pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:47.011968 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.011087    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b84ba29-a116-48ef-8d9f-d81ceb4b3359-metrics-certs\") pod \"network-metrics-daemon-6xvgt\" (UID: \"4b84ba29-a116-48ef-8d9f-d81ceb4b3359\") " pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:47.037249 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.037212    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"
Nov 03 21:14:47.039400 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.039385    5053 scope.go:122] "RemoveContainer" containerID="9f6b6dc724c5973a7c58dc058c00d7b81472be8fbb45b2fedb085ed90d0ec098"
Nov 03 21:14:47.042556 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.042535    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-j2ksl"
Nov 03 21:14:47.048305 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.048287    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xvgt"
Nov 03 21:14:47.049625 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.049608    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:47.053089 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.053070    5053 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:14:47.066254 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.066215    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:47.178980 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.178952    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-554sk_3d34868b-605a-4bfb-81a3-432e0db0ce6a/dns-node-resolver/0.log"
Nov 03 21:14:47.439893 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.439817    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz"]
Nov 03 21:14:47.485965 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.485941    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-j2ksl"]
Nov 03 21:14:47.497423 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:47.497373    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8a800c_36be_4da0_b1e6_5ef65f994fc1.slice/crio-908844e1c31e79469632e6b912cee9703812c12fceb13fa2c7d8e19e4b30d460 WatchSource:0}: Error finding container 908844e1c31e79469632e6b912cee9703812c12fceb13fa2c7d8e19e4b30d460: Status 404 returned error can't find the container with id 908844e1c31e79469632e6b912cee9703812c12fceb13fa2c7d8e19e4b30d460
Nov 03 21:14:47.514187 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.514165    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:47.514338 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.514200    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:47.514400 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.514345    5053 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 21:14:47.514400 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.514358    5053 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75cb56744b-c6rsx: secret "image-registry-tls" not found
Nov 03 21:14:47.514400 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.514378    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jf555\" (UID: \"ae204c86-8d93-4adf-988c-cd836e011270\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:47.514400 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.514399    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls podName:ef7a859e-70f5-497a-9ad4-6614885fb71e nodeName:}" failed. No retries permitted until 2026-11-03 21:14:48.514387584 +0000 UTC m=+3.008309437 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls") pod "image-registry-75cb56744b-c6rsx" (UID: "ef7a859e-70f5-497a-9ad4-6614885fb71e") : secret "image-registry-tls" not found
Nov 03 21:14:47.514542 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.514347    5053 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 21:14:47.514542 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.514446    5053 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6df897dfcf-2b92c: secret "image-registry-tls" not found
Nov 03 21:14:47.514542 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.514447    5053 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 21:14:47.514542 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.514486    5053 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 21:14:47.514542 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.514447    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:47.514542 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.514497    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls podName:1db07cc2-aa71-490e-a21a-0a8cb06d94d9 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:48.514485859 +0000 UTC m=+3.008407714 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls") pod "image-registry-6df897dfcf-2b92c" (UID: "1db07cc2-aa71-490e-a21a-0a8cb06d94d9") : secret "image-registry-tls" not found
Nov 03 21:14:47.514542 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.514525    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls podName:cad3df5d-3fe9-449b-8b70-793bceb47985 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:48.51451098 +0000 UTC m=+3.008432839 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9vxl" (UID: "cad3df5d-3fe9-449b-8b70-793bceb47985") : secret "insights-runtime-extractor-tls" not found
Nov 03 21:14:47.514542 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.514541    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls podName:ae204c86-8d93-4adf-988c-cd836e011270 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:48.514532903 +0000 UTC m=+3.008454760 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jf555" (UID: "ae204c86-8d93-4adf-988c-cd836e011270") : secret "cluster-monitoring-operator-tls" not found
Nov 03 21:14:47.514801 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.514603    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:47.514801 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.514658    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:47.514801 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.514709    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle podName:7ccca940-de12-451b-b920-d6bef516942f nodeName:}" failed. No retries permitted until 2026-11-03 21:14:48.514701466 +0000 UTC m=+3.008623321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle") pod "router-default-5ffbbbfcf7-99qbj" (UID: "7ccca940-de12-451b-b920-d6bef516942f") : configmap references non-existent config key: service-ca.crt
Nov 03 21:14:47.514801 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.514740    5053 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 21:14:47.514801 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.514778    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs podName:7ccca940-de12-451b-b920-d6bef516942f nodeName:}" failed. No retries permitted until 2026-11-03 21:14:48.514768773 +0000 UTC m=+3.008690632 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs") pod "router-default-5ffbbbfcf7-99qbj" (UID: "7ccca940-de12-451b-b920-d6bef516942f") : secret "router-metrics-certs-default" not found
Nov 03 21:14:47.549037 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.548999    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6xvgt"]
Nov 03 21:14:47.553381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.553358    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8p8hk"]
Nov 03 21:14:47.555695 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:47.555672    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b84ba29_a116_48ef_8d9f_d81ceb4b3359.slice/crio-fdba6dc3199ee50ffca6f0245d25417831d68e0d80c7bed8641f5787516ef882 WatchSource:0}: Error finding container fdba6dc3199ee50ffca6f0245d25417831d68e0d80c7bed8641f5787516ef882: Status 404 returned error can't find the container with id fdba6dc3199ee50ffca6f0245d25417831d68e0d80c7bed8641f5787516ef882
Nov 03 21:14:47.556461 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:47.556440    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaadb342f_3310_4c81_967d_0f0a8ef5a4e6.slice/crio-fb7449b618c49ad77a887363cf1d4f2e22b690651aea00c51f169f9971c6b7f3 WatchSource:0}: Error finding container fb7449b618c49ad77a887363cf1d4f2e22b690651aea00c51f169f9971c6b7f3: Status 404 returned error can't find the container with id fb7449b618c49ad77a887363cf1d4f2e22b690651aea00c51f169f9971c6b7f3
Nov 03 21:14:47.771934 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.771896    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz" event={"ID":"cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c","Type":"ContainerStarted","Data":"5535c1b360a6d378b4656e079dc8db19953d1e3e878665c8af5a90f93c9aa60b"}
Nov 03 21:14:47.773567 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.773548    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 21:14:47.773948 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.773931    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/1.log"
Nov 03 21:14:47.774009 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.773966    5053 generic.go:356] "Generic (PLEG): container finished" podID="6547c71f-71a2-40c9-912f-9283b6bdb560" containerID="63cad6d81e80240e0b18d92dd9fc013289b0312ddf594bb33a2dbe34515f84d7" exitCode=255
Nov 03 21:14:47.774066 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.774046    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv" event={"ID":"6547c71f-71a2-40c9-912f-9283b6bdb560","Type":"ContainerDied","Data":"63cad6d81e80240e0b18d92dd9fc013289b0312ddf594bb33a2dbe34515f84d7"}
Nov 03 21:14:47.774132 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.774117    5053 scope.go:122] "RemoveContainer" containerID="9f6b6dc724c5973a7c58dc058c00d7b81472be8fbb45b2fedb085ed90d0ec098"
Nov 03 21:14:47.776660 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.774413    5053 scope.go:122] "RemoveContainer" containerID="63cad6d81e80240e0b18d92dd9fc013289b0312ddf594bb33a2dbe34515f84d7"
Nov 03 21:14:47.776660 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:47.774634    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rhpjv_openshift-console-operator(6547c71f-71a2-40c9-912f-9283b6bdb560)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv" podUID="6547c71f-71a2-40c9-912f-9283b6bdb560"
Nov 03 21:14:47.778355 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.778335    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-j2ksl" event={"ID":"bb8a800c-36be-4da0-b1e6-5ef65f994fc1","Type":"ContainerStarted","Data":"908844e1c31e79469632e6b912cee9703812c12fceb13fa2c7d8e19e4b30d460"}
Nov 03 21:14:47.779501 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.779483    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6xvgt" event={"ID":"4b84ba29-a116-48ef-8d9f-d81ceb4b3359","Type":"ContainerStarted","Data":"fdba6dc3199ee50ffca6f0245d25417831d68e0d80c7bed8641f5787516ef882"}
Nov 03 21:14:47.780443 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:47.780424    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8p8hk" event={"ID":"aadb342f-3310-4c81-967d-0f0a8ef5a4e6","Type":"ContainerStarted","Data":"fb7449b618c49ad77a887363cf1d4f2e22b690651aea00c51f169f9971c6b7f3"}
Nov 03 21:14:48.007835 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:48.007810    5053 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:48.523183 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:48.523149    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:48.523407 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:48.523190    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jf555\" (UID: \"ae204c86-8d93-4adf-988c-cd836e011270\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:48.523407 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:48.523260    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:48.523407 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:48.523298    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:48.523407 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:48.523328    5053 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 21:14:48.523407 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:48.523348    5053 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75cb56744b-c6rsx: secret "image-registry-tls" not found
Nov 03 21:14:48.523407 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:48.523351    5053 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 21:14:48.523407 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:48.523382    5053 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 21:14:48.523407 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:48.523333    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:48.523407 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:48.523414    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls podName:ef7a859e-70f5-497a-9ad4-6614885fb71e nodeName:}" failed. No retries permitted until 2026-11-03 21:14:50.523396054 +0000 UTC m=+5.017317909 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls") pod "image-registry-75cb56744b-c6rsx" (UID: "ef7a859e-70f5-497a-9ad4-6614885fb71e") : secret "image-registry-tls" not found
Nov 03 21:14:48.523841 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:48.523442    5053 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 21:14:48.523841 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:48.523445    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls podName:ae204c86-8d93-4adf-988c-cd836e011270 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:50.523436541 +0000 UTC m=+5.017358402 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jf555" (UID: "ae204c86-8d93-4adf-988c-cd836e011270") : secret "cluster-monitoring-operator-tls" not found
Nov 03 21:14:48.523841 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:48.523462    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs podName:7ccca940-de12-451b-b920-d6bef516942f nodeName:}" failed. No retries permitted until 2026-11-03 21:14:50.523454092 +0000 UTC m=+5.017375949 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs") pod "router-default-5ffbbbfcf7-99qbj" (UID: "7ccca940-de12-451b-b920-d6bef516942f") : secret "router-metrics-certs-default" not found
Nov 03 21:14:48.523841 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:48.523478    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls podName:cad3df5d-3fe9-449b-8b70-793bceb47985 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:50.523469785 +0000 UTC m=+5.017391648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9vxl" (UID: "cad3df5d-3fe9-449b-8b70-793bceb47985") : secret "insights-runtime-extractor-tls" not found
Nov 03 21:14:48.523841 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:48.523498    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle podName:7ccca940-de12-451b-b920-d6bef516942f nodeName:}" failed. No retries permitted until 2026-11-03 21:14:50.523489281 +0000 UTC m=+5.017411142 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle") pod "router-default-5ffbbbfcf7-99qbj" (UID: "7ccca940-de12-451b-b920-d6bef516942f") : configmap references non-existent config key: service-ca.crt
Nov 03 21:14:48.523841 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:48.523550    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:48.523841 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:48.523658    5053 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 21:14:48.523841 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:48.523670    5053 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6df897dfcf-2b92c: secret "image-registry-tls" not found
Nov 03 21:14:48.523841 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:48.523711    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls podName:1db07cc2-aa71-490e-a21a-0a8cb06d94d9 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:50.523699936 +0000 UTC m=+5.017621792 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls") pod "image-registry-6df897dfcf-2b92c" (UID: "1db07cc2-aa71-490e-a21a-0a8cb06d94d9") : secret "image-registry-tls" not found
Nov 03 21:14:48.575325 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:48.575296    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z5s4n_c3560fa9-2666-4e0d-854c-3c126489c61d/node-ca/0.log"
Nov 03 21:14:48.789595 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:48.789506    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v6qdv" event={"ID":"356632db-499f-4ced-bf01-b280472b53f1","Type":"ContainerStarted","Data":"cdf1143ab097f1fa1a4054b5bde2b5a77c16753008f0fbc02a6ace74540d3702"}
Nov 03 21:14:48.792448 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:48.792418    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 21:14:48.792933 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:48.792914    5053 scope.go:122] "RemoveContainer" containerID="63cad6d81e80240e0b18d92dd9fc013289b0312ddf594bb33a2dbe34515f84d7"
Nov 03 21:14:48.793217 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:48.793191    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rhpjv_openshift-console-operator(6547c71f-71a2-40c9-912f-9283b6bdb560)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv" podUID="6547c71f-71a2-40c9-912f-9283b6bdb560"
Nov 03 21:14:48.796056 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:48.796016    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jqkbh" event={"ID":"2000161a-2aa8-4638-a4f7-fbe305e929ae","Type":"ContainerStarted","Data":"6d62c3f06d0c603197b563a657d10eec3a869e4a33f4e67f92851686542e3a6c"}
Nov 03 21:14:48.973533 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:48.973397    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m5s9b_c27ff312-c8d3-4691-b535-ea8adf44d9c3/serve-healthcheck-canary/0.log"
Nov 03 21:14:49.173305 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:49.173202    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-jqkbh_2000161a-2aa8-4638-a4f7-fbe305e929ae/migrator/0.log"
Nov 03 21:14:49.372504 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:49.372469    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-v6qdv_356632db-499f-4ced-bf01-b280472b53f1/migrator/0.log"
Nov 03 21:14:49.579295 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:49.579266    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7_eea6e5c2-57f8-42c6-9ce9-55a3c8f92790/kube-storage-version-migrator-operator/0.log"
Nov 03 21:14:50.552541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:50.552504    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:50.552970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:50.552553    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:50.552970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:50.552584    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:50.552970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:50.552602    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:50.552970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:50.552621    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:50.552970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:50.552639    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jf555\" (UID: \"ae204c86-8d93-4adf-988c-cd836e011270\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:50.552970 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:50.552646    5053 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 21:14:50.552970 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:50.552719    5053 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 21:14:50.552970 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:50.552728    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls podName:cad3df5d-3fe9-449b-8b70-793bceb47985 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:54.552708255 +0000 UTC m=+9.046630131 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9vxl" (UID: "cad3df5d-3fe9-449b-8b70-793bceb47985") : secret "insights-runtime-extractor-tls" not found
Nov 03 21:14:50.552970 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:50.552802    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs podName:7ccca940-de12-451b-b920-d6bef516942f nodeName:}" failed. No retries permitted until 2026-11-03 21:14:54.552777991 +0000 UTC m=+9.046699845 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs") pod "router-default-5ffbbbfcf7-99qbj" (UID: "7ccca940-de12-451b-b920-d6bef516942f") : secret "router-metrics-certs-default" not found
Nov 03 21:14:50.552970 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:50.552821    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle podName:7ccca940-de12-451b-b920-d6bef516942f nodeName:}" failed. No retries permitted until 2026-11-03 21:14:54.552810673 +0000 UTC m=+9.046732546 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle") pod "router-default-5ffbbbfcf7-99qbj" (UID: "7ccca940-de12-451b-b920-d6bef516942f") : configmap references non-existent config key: service-ca.crt
Nov 03 21:14:50.555265 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:50.555209    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls\") pod \"image-registry-75cb56744b-c6rsx\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") " pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:50.555265 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:50.555258    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae204c86-8d93-4adf-988c-cd836e011270-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jf555\" (UID: \"ae204c86-8d93-4adf-988c-cd836e011270\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:50.555408 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:50.555304    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls\") pod \"image-registry-6df897dfcf-2b92c\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") " pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:50.636001 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:50.635972    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"
Nov 03 21:14:50.643292 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:50.643269    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:50.649439 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:50.649415    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:50.805804 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:50.805715    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-j2ksl" event={"ID":"bb8a800c-36be-4da0-b1e6-5ef65f994fc1","Type":"ContainerStarted","Data":"bdddb3a824faa8c15bcb4e2d2b911c7232a4a18fbe0563fe5db2ccb4f3d53007"}
Nov 03 21:14:50.807590 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:50.807566    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6xvgt" event={"ID":"4b84ba29-a116-48ef-8d9f-d81ceb4b3359","Type":"ContainerStarted","Data":"c671c35b9ef2a54bf13b508a0f3a380c0b21f9febd42899958a671117a09c6c6"}
Nov 03 21:14:50.807716 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:50.807606    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6xvgt" event={"ID":"4b84ba29-a116-48ef-8d9f-d81ceb4b3359","Type":"ContainerStarted","Data":"dd17d0b541d55baa541ce3a9882d8fe22941d9d6f02b587666244ea733b66620"}
Nov 03 21:14:51.269408 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:51.269380    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:14:51.469375 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:51.469345    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-jf555"]
Nov 03 21:14:51.513317 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:51.513281    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae204c86_8d93_4adf_988c_cd836e011270.slice/crio-53f7fba0020f8790dda29be9c3481b301f634602f897c7adc9e57ae992f2d1f5 WatchSource:0}: Error finding container 53f7fba0020f8790dda29be9c3481b301f634602f897c7adc9e57ae992f2d1f5: Status 404 returned error can't find the container with id 53f7fba0020f8790dda29be9c3481b301f634602f897c7adc9e57ae992f2d1f5
Nov 03 21:14:51.526569 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:51.526544    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75cb56744b-c6rsx"]
Nov 03 21:14:51.529849 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:51.529810    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7a859e_70f5_497a_9ad4_6614885fb71e.slice/crio-9ea91149f669e1acb18d4a39de853703caee922ca58e77977727b5fa889aaeb4 WatchSource:0}: Error finding container 9ea91149f669e1acb18d4a39de853703caee922ca58e77977727b5fa889aaeb4: Status 404 returned error can't find the container with id 9ea91149f669e1acb18d4a39de853703caee922ca58e77977727b5fa889aaeb4
Nov 03 21:14:51.572283 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:51.572192    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6df897dfcf-2b92c"]
Nov 03 21:14:51.579464 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:14:51.579439    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db07cc2_aa71_490e_a21a_0a8cb06d94d9.slice/crio-78fdf90155ed86aec2073837c54cde87dd033c496fecac9e4619cb4177bb84fd WatchSource:0}: Error finding container 78fdf90155ed86aec2073837c54cde87dd033c496fecac9e4619cb4177bb84fd: Status 404 returned error can't find the container with id 78fdf90155ed86aec2073837c54cde87dd033c496fecac9e4619cb4177bb84fd
Nov 03 21:14:51.816950 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:51.816851    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8p8hk" event={"ID":"aadb342f-3310-4c81-967d-0f0a8ef5a4e6","Type":"ContainerStarted","Data":"7ace9cb731a15d67621c7525a25762b0fe78bde6feb5b2ab5cfba94fa821f416"}
Nov 03 21:14:51.816950 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:51.816905    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8p8hk" event={"ID":"aadb342f-3310-4c81-967d-0f0a8ef5a4e6","Type":"ContainerStarted","Data":"52a374e25da5c9f663e1d849b2c9ee4a58d65f841842fb7089fbb1c42562ead6"}
Nov 03 21:14:51.816950 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:51.816945    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:14:51.818893 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:51.818861    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz" event={"ID":"cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c","Type":"ContainerStarted","Data":"912bb897cbcf00ad275225a33c7ef810f4470f1992eeafbfe9436d8725876f46"}
Nov 03 21:14:51.819030 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:51.818917    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqlqz" event={"ID":"cbc3e9cb-524f-4ad4-9a4a-c3510f25ac9c","Type":"ContainerStarted","Data":"8661d28e724b4d824295ceed65ebfd5f22040e2720ba05c6e4e948ff971012a3"}
Nov 03 21:14:51.820201 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:51.820180    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555" event={"ID":"ae204c86-8d93-4adf-988c-cd836e011270","Type":"ContainerStarted","Data":"53f7fba0020f8790dda29be9c3481b301f634602f897c7adc9e57ae992f2d1f5"}
Nov 03 21:14:51.821929 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:51.821902    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75cb56744b-c6rsx" event={"ID":"ef7a859e-70f5-497a-9ad4-6614885fb71e","Type":"ContainerStarted","Data":"d84f3f05da84fdf8843c1da4fc34c3f0153792c25ddf218ccb92a6c9b927b92a"}
Nov 03 21:14:51.822043 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:51.821934    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75cb56744b-c6rsx" event={"ID":"ef7a859e-70f5-497a-9ad4-6614885fb71e","Type":"ContainerStarted","Data":"9ea91149f669e1acb18d4a39de853703caee922ca58e77977727b5fa889aaeb4"}
Nov 03 21:14:51.823602 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:51.823581    5053 generic.go:356] "Generic (PLEG): container finished" podID="eea6e5c2-57f8-42c6-9ce9-55a3c8f92790" containerID="67117625ad30e140b253bdd2bc78373b0e1011f190d089c545d45d2c350f6500" exitCode=0
Nov 03 21:14:51.823713 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:51.823653    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7" event={"ID":"eea6e5c2-57f8-42c6-9ce9-55a3c8f92790","Type":"ContainerDied","Data":"67117625ad30e140b253bdd2bc78373b0e1011f190d089c545d45d2c350f6500"}
Nov 03 21:14:51.824007 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:51.823989    5053 scope.go:122] "RemoveContainer" containerID="67117625ad30e140b253bdd2bc78373b0e1011f190d089c545d45d2c350f6500"
Nov 03 21:14:51.826318 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:51.826292    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6df897dfcf-2b92c" event={"ID":"1db07cc2-aa71-490e-a21a-0a8cb06d94d9","Type":"ContainerStarted","Data":"35a3a7301caceb4f3a2db88db71fb7cf2aa32666554ee3abe83920847084c6d2"}
Nov 03 21:14:51.826437 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:51.826337    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6df897dfcf-2b92c" event={"ID":"1db07cc2-aa71-490e-a21a-0a8cb06d94d9","Type":"ContainerStarted","Data":"78fdf90155ed86aec2073837c54cde87dd033c496fecac9e4619cb4177bb84fd"}
Nov 03 21:14:52.832900 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:52.832869    5053 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 21:14:52.833440 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:52.832930    5053 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 21:14:52.833440 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:52.832913    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tbdq7" event={"ID":"eea6e5c2-57f8-42c6-9ce9-55a3c8f92790","Type":"ContainerStarted","Data":"8fcc6adb80efda74b919782f819d5239f832d1ff3dfe6d237b2c44bad6a6dd2d"}
Nov 03 21:14:53.710666 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:53.710637    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:14:53.710951 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:53.710932    5053 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 21:14:53.838558 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:53.838529    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jf555" event={"ID":"ae204c86-8d93-4adf-988c-cd836e011270","Type":"ContainerStarted","Data":"41996ddbb79e9e8b91de067f0c0dc82d80101f360915b9690ea06a264017d555"}
Nov 03 21:14:54.372535 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:54.372508    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:54.372903 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:54.372889    5053 scope.go:122] "RemoveContainer" containerID="63cad6d81e80240e0b18d92dd9fc013289b0312ddf594bb33a2dbe34515f84d7"
Nov 03 21:14:54.373088 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:54.373071    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rhpjv_openshift-console-operator(6547c71f-71a2-40c9-912f-9283b6bdb560)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv" podUID="6547c71f-71a2-40c9-912f-9283b6bdb560"
Nov 03 21:14:54.591550 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:54.591511    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:54.591751 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:54.591567    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:54.591751 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:54.591613    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:14:54.591751 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:54.591698    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle podName:7ccca940-de12-451b-b920-d6bef516942f nodeName:}" failed. No retries permitted until 2026-11-03 21:15:02.591675305 +0000 UTC m=+17.085597185 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle") pod "router-default-5ffbbbfcf7-99qbj" (UID: "7ccca940-de12-451b-b920-d6bef516942f") : configmap references non-existent config key: service-ca.crt
Nov 03 21:14:54.591751 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:54.591706    5053 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 21:14:54.591751 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:14:54.591752    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls podName:cad3df5d-3fe9-449b-8b70-793bceb47985 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:02.591742587 +0000 UTC m=+17.085664459 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9vxl" (UID: "cad3df5d-3fe9-449b-8b70-793bceb47985") : secret "insights-runtime-extractor-tls" not found
Nov 03 21:14:54.594076 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:54.594055    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ccca940-de12-451b-b920-d6bef516942f-metrics-certs\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:14:55.417447 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:55.417399    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:14:55.419913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:55.419895    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:14:58.659977 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:58.659946    5053 scope.go:122] "RemoveContainer" containerID="63cad6d81e80240e0b18d92dd9fc013289b0312ddf594bb33a2dbe34515f84d7"
Nov 03 21:14:58.860107 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:58.860078    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 21:14:58.860282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:58.860152    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv" event={"ID":"6547c71f-71a2-40c9-912f-9283b6bdb560","Type":"ContainerStarted","Data":"47e1e8865739eec98e108562d316fb77f6d2d072e3757b5bfb96a74ef81501a5"}
Nov 03 21:14:58.860465 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:58.860448    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:14:58.861570 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:58.861545    5053 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-rhpjv container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.16:8443/readyz\": dial tcp 10.132.0.16:8443: connect: connection refused" start-of-body=
Nov 03 21:14:58.861654 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:58.861596    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv" podUID="6547c71f-71a2-40c9-912f-9283b6bdb560" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.16:8443/readyz\": dial tcp 10.132.0.16:8443: connect: connection refused"
Nov 03 21:14:59.868392 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:14:59.868362    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rhpjv"
Nov 03 21:15:00.958166 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:00.958140    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvkn6"
Nov 03 21:15:01.835383 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:01.835356    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8p8hk"
Nov 03 21:15:02.660900 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:02.660863    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:15:02.661436 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:02.660932    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:15:02.662070 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:02.662040    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ccca940-de12-451b-b920-d6bef516942f-service-ca-bundle\") pod \"router-default-5ffbbbfcf7-99qbj\" (UID: \"7ccca940-de12-451b-b920-d6bef516942f\") " pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:15:02.663588 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:02.663567    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cad3df5d-3fe9-449b-8b70-793bceb47985-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9vxl\" (UID: \"cad3df5d-3fe9-449b-8b70-793bceb47985\") " pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:15:02.938093 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:02.937985    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:15:02.951132 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:02.951109    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v9vxl"
Nov 03 21:15:03.411443 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:03.411407    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v9vxl"]
Nov 03 21:15:03.414358 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:15:03.414331    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad3df5d_3fe9_449b_8b70_793bceb47985.slice/crio-ad8ed14caf47e40f1942fc28e53a42a9d9ba553e4257fcfc03fd5896c21e259d WatchSource:0}: Error finding container ad8ed14caf47e40f1942fc28e53a42a9d9ba553e4257fcfc03fd5896c21e259d: Status 404 returned error can't find the container with id ad8ed14caf47e40f1942fc28e53a42a9d9ba553e4257fcfc03fd5896c21e259d
Nov 03 21:15:03.461182 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:03.461146    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5ffbbbfcf7-99qbj"]
Nov 03 21:15:03.464521 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:15:03.464495    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ccca940_de12_451b_b920_d6bef516942f.slice/crio-5fe4c3e6e586e257bf41cac6206a0efc48c49975dbf8e29f6b265e7e4ea96f23 WatchSource:0}: Error finding container 5fe4c3e6e586e257bf41cac6206a0efc48c49975dbf8e29f6b265e7e4ea96f23: Status 404 returned error can't find the container with id 5fe4c3e6e586e257bf41cac6206a0efc48c49975dbf8e29f6b265e7e4ea96f23
Nov 03 21:15:03.881578 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:03.881534    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v9vxl" event={"ID":"cad3df5d-3fe9-449b-8b70-793bceb47985","Type":"ContainerStarted","Data":"be90d8e451ca25178f054ec0c13af7073d7dd6ea5a98ff4b7d2ff5c346a1ab51"}
Nov 03 21:15:03.882016 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:03.881597    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v9vxl" event={"ID":"cad3df5d-3fe9-449b-8b70-793bceb47985","Type":"ContainerStarted","Data":"ad8ed14caf47e40f1942fc28e53a42a9d9ba553e4257fcfc03fd5896c21e259d"}
Nov 03 21:15:03.883115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:03.883089    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj" event={"ID":"7ccca940-de12-451b-b920-d6bef516942f","Type":"ContainerStarted","Data":"e4c8d87c8aa0db2a3646da4a75f6a6c2fb8a3b9c3e395afef9cb9285c9749a0d"}
Nov 03 21:15:03.883224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:03.883128    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj" event={"ID":"7ccca940-de12-451b-b920-d6bef516942f","Type":"ContainerStarted","Data":"5fe4c3e6e586e257bf41cac6206a0efc48c49975dbf8e29f6b265e7e4ea96f23"}
Nov 03 21:15:03.938748 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:03.938724    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:15:03.941226 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:03.941188    5053 patch_prober.go:29] interesting pod/router-default-5ffbbbfcf7-99qbj container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld
Nov 03 21:15:03.941226 ip-10-0-128-43 kubenswrapper[5053]: [-]has-synced failed: reason withheld
Nov 03 21:15:03.941226 ip-10-0-128-43 kubenswrapper[5053]: [+]process-running ok
Nov 03 21:15:03.941226 ip-10-0-128-43 kubenswrapper[5053]: healthz check failed
Nov 03 21:15:03.941459 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:03.941264    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj" podUID="7ccca940-de12-451b-b920-d6bef516942f" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Nov 03 21:15:04.312020 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.311986    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-f7zn9"]
Nov 03 21:15:04.314754 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.314718    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s4x67"]
Nov 03 21:15:04.314892 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.314859    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-f7zn9"
Nov 03 21:15:04.316987 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.316966    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s4x67"
Nov 03 21:15:04.334730 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.334701    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6df897dfcf-2b92c"]
Nov 03 21:15:04.372491 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.372445    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 21:15:04.372491 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.372478    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 21:15:04.372491 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.372493    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-fzzv4\""
Nov 03 21:15:04.372768 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.372537    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-pdppx\""
Nov 03 21:15:04.372768 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.372540    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 03 21:15:04.374131 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.374107    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a77b27d-5a39-436f-b09d-21872b22709d-tmp\") pod \"downloads-6dd789c8b7-f7zn9\" (UID: \"7a77b27d-5a39-436f-b09d-21872b22709d\") " pod="openshift-console/downloads-6dd789c8b7-f7zn9"
Nov 03 21:15:04.374224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.374147    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5bb51b13-c088-4035-bd9e-91c74dfe3a95-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-s4x67\" (UID: \"5bb51b13-c088-4035-bd9e-91c74dfe3a95\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s4x67"
Nov 03 21:15:04.374353 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.374331    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g8dk\" (UniqueName: \"kubernetes.io/projected/7a77b27d-5a39-436f-b09d-21872b22709d-kube-api-access-6g8dk\") pod \"downloads-6dd789c8b7-f7zn9\" (UID: \"7a77b27d-5a39-436f-b09d-21872b22709d\") " pod="openshift-console/downloads-6dd789c8b7-f7zn9"
Nov 03 21:15:04.410630 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.410585    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-854fddcd9b-7zs44"]
Nov 03 21:15:04.415919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.415067    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.469077 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.469043    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s4x67"]
Nov 03 21:15:04.474439 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.474407    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-854fddcd9b-7zs44"]
Nov 03 21:15:04.474873 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.474833    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a77b27d-5a39-436f-b09d-21872b22709d-tmp\") pod \"downloads-6dd789c8b7-f7zn9\" (UID: \"7a77b27d-5a39-436f-b09d-21872b22709d\") " pod="openshift-console/downloads-6dd789c8b7-f7zn9"
Nov 03 21:15:04.474978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.474886    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5bb51b13-c088-4035-bd9e-91c74dfe3a95-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-s4x67\" (UID: \"5bb51b13-c088-4035-bd9e-91c74dfe3a95\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s4x67"
Nov 03 21:15:04.474978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.474925    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d637e4ed-6bbd-44f1-b091-4594db93e2d8-installation-pull-secrets\") pod \"image-registry-854fddcd9b-7zs44\" (UID: \"d637e4ed-6bbd-44f1-b091-4594db93e2d8\") " pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.474978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.474951    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdktj\" (UniqueName: \"kubernetes.io/projected/d637e4ed-6bbd-44f1-b091-4594db93e2d8-kube-api-access-jdktj\") pod \"image-registry-854fddcd9b-7zs44\" (UID: \"d637e4ed-6bbd-44f1-b091-4594db93e2d8\") " pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.475131 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.474995    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d637e4ed-6bbd-44f1-b091-4594db93e2d8-trusted-ca\") pod \"image-registry-854fddcd9b-7zs44\" (UID: \"d637e4ed-6bbd-44f1-b091-4594db93e2d8\") " pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.475131 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.475034    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d637e4ed-6bbd-44f1-b091-4594db93e2d8-registry-tls\") pod \"image-registry-854fddcd9b-7zs44\" (UID: \"d637e4ed-6bbd-44f1-b091-4594db93e2d8\") " pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.475131 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.475076    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d637e4ed-6bbd-44f1-b091-4594db93e2d8-registry-certificates\") pod \"image-registry-854fddcd9b-7zs44\" (UID: \"d637e4ed-6bbd-44f1-b091-4594db93e2d8\") " pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.475131 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.475094    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d637e4ed-6bbd-44f1-b091-4594db93e2d8-ca-trust-extracted\") pod \"image-registry-854fddcd9b-7zs44\" (UID: \"d637e4ed-6bbd-44f1-b091-4594db93e2d8\") " pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.475131 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.475121    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g8dk\" (UniqueName: \"kubernetes.io/projected/7a77b27d-5a39-436f-b09d-21872b22709d-kube-api-access-6g8dk\") pod \"downloads-6dd789c8b7-f7zn9\" (UID: \"7a77b27d-5a39-436f-b09d-21872b22709d\") " pod="openshift-console/downloads-6dd789c8b7-f7zn9"
Nov 03 21:15:04.475411 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.475147    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d637e4ed-6bbd-44f1-b091-4594db93e2d8-image-registry-private-configuration\") pod \"image-registry-854fddcd9b-7zs44\" (UID: \"d637e4ed-6bbd-44f1-b091-4594db93e2d8\") " pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.475411 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.475181    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d637e4ed-6bbd-44f1-b091-4594db93e2d8-bound-sa-token\") pod \"image-registry-854fddcd9b-7zs44\" (UID: \"d637e4ed-6bbd-44f1-b091-4594db93e2d8\") " pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.475411 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.475364    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a77b27d-5a39-436f-b09d-21872b22709d-tmp\") pod \"downloads-6dd789c8b7-f7zn9\" (UID: \"7a77b27d-5a39-436f-b09d-21872b22709d\") " pod="openshift-console/downloads-6dd789c8b7-f7zn9"
Nov 03 21:15:04.478461 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.478419    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5bb51b13-c088-4035-bd9e-91c74dfe3a95-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-s4x67\" (UID: \"5bb51b13-c088-4035-bd9e-91c74dfe3a95\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s4x67"
Nov 03 21:15:04.533495 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.533448    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-f7zn9"]
Nov 03 21:15:04.539784 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.539752    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g8dk\" (UniqueName: \"kubernetes.io/projected/7a77b27d-5a39-436f-b09d-21872b22709d-kube-api-access-6g8dk\") pod \"downloads-6dd789c8b7-f7zn9\" (UID: \"7a77b27d-5a39-436f-b09d-21872b22709d\") " pod="openshift-console/downloads-6dd789c8b7-f7zn9"
Nov 03 21:15:04.576542 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.576510    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d637e4ed-6bbd-44f1-b091-4594db93e2d8-registry-certificates\") pod \"image-registry-854fddcd9b-7zs44\" (UID: \"d637e4ed-6bbd-44f1-b091-4594db93e2d8\") " pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.576542 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.576545    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d637e4ed-6bbd-44f1-b091-4594db93e2d8-ca-trust-extracted\") pod \"image-registry-854fddcd9b-7zs44\" (UID: \"d637e4ed-6bbd-44f1-b091-4594db93e2d8\") " pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.576696 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.576574    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d637e4ed-6bbd-44f1-b091-4594db93e2d8-image-registry-private-configuration\") pod \"image-registry-854fddcd9b-7zs44\" (UID: \"d637e4ed-6bbd-44f1-b091-4594db93e2d8\") " pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.576696 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.576604    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d637e4ed-6bbd-44f1-b091-4594db93e2d8-bound-sa-token\") pod \"image-registry-854fddcd9b-7zs44\" (UID: \"d637e4ed-6bbd-44f1-b091-4594db93e2d8\") " pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.576696 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.576659    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d637e4ed-6bbd-44f1-b091-4594db93e2d8-installation-pull-secrets\") pod \"image-registry-854fddcd9b-7zs44\" (UID: \"d637e4ed-6bbd-44f1-b091-4594db93e2d8\") " pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.576696 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.576683    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdktj\" (UniqueName: \"kubernetes.io/projected/d637e4ed-6bbd-44f1-b091-4594db93e2d8-kube-api-access-jdktj\") pod \"image-registry-854fddcd9b-7zs44\" (UID: \"d637e4ed-6bbd-44f1-b091-4594db93e2d8\") " pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.576860 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.576715    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d637e4ed-6bbd-44f1-b091-4594db93e2d8-trusted-ca\") pod \"image-registry-854fddcd9b-7zs44\" (UID: \"d637e4ed-6bbd-44f1-b091-4594db93e2d8\") " pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.576860 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.576749    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d637e4ed-6bbd-44f1-b091-4594db93e2d8-registry-tls\") pod \"image-registry-854fddcd9b-7zs44\" (UID: \"d637e4ed-6bbd-44f1-b091-4594db93e2d8\") " pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.576967 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.576944    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d637e4ed-6bbd-44f1-b091-4594db93e2d8-ca-trust-extracted\") pod \"image-registry-854fddcd9b-7zs44\" (UID: \"d637e4ed-6bbd-44f1-b091-4594db93e2d8\") " pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.577750 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.577729    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d637e4ed-6bbd-44f1-b091-4594db93e2d8-trusted-ca\") pod \"image-registry-854fddcd9b-7zs44\" (UID: \"d637e4ed-6bbd-44f1-b091-4594db93e2d8\") " pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.578041 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.578021    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d637e4ed-6bbd-44f1-b091-4594db93e2d8-registry-certificates\") pod \"image-registry-854fddcd9b-7zs44\" (UID: \"d637e4ed-6bbd-44f1-b091-4594db93e2d8\") " pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.579283 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.579264    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d637e4ed-6bbd-44f1-b091-4594db93e2d8-image-registry-private-configuration\") pod \"image-registry-854fddcd9b-7zs44\" (UID: \"d637e4ed-6bbd-44f1-b091-4594db93e2d8\") " pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.579855 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.579835    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d637e4ed-6bbd-44f1-b091-4594db93e2d8-registry-tls\") pod \"image-registry-854fddcd9b-7zs44\" (UID: \"d637e4ed-6bbd-44f1-b091-4594db93e2d8\") " pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.579966 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.579943    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d637e4ed-6bbd-44f1-b091-4594db93e2d8-installation-pull-secrets\") pod \"image-registry-854fddcd9b-7zs44\" (UID: \"d637e4ed-6bbd-44f1-b091-4594db93e2d8\") " pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.628840 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.628813    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-f7zn9"
Nov 03 21:15:04.633373 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.633170    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s4x67"
Nov 03 21:15:04.633598 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.633568    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdktj\" (UniqueName: \"kubernetes.io/projected/d637e4ed-6bbd-44f1-b091-4594db93e2d8-kube-api-access-jdktj\") pod \"image-registry-854fddcd9b-7zs44\" (UID: \"d637e4ed-6bbd-44f1-b091-4594db93e2d8\") " pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.633841 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.633822    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d637e4ed-6bbd-44f1-b091-4594db93e2d8-bound-sa-token\") pod \"image-registry-854fddcd9b-7zs44\" (UID: \"d637e4ed-6bbd-44f1-b091-4594db93e2d8\") " pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.728033 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.727986    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:04.889346 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.889221    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v9vxl" event={"ID":"cad3df5d-3fe9-449b-8b70-793bceb47985","Type":"ContainerStarted","Data":"9f5ab9705edba30d999e9e526347872fd2c7db31c74bdbd9201485338d06ca9e"}
Nov 03 21:15:04.940229 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:04.940199    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:15:05.064795 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:05.064684    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s4x67"]
Nov 03 21:15:05.065814 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:05.065716    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-f7zn9"]
Nov 03 21:15:05.068333 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:15:05.068309    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a77b27d_5a39_436f_b09d_21872b22709d.slice/crio-71ea15c6966b2924a5b991f5d4707a2a518f53314f73eecafa98ede1832fe5f1 WatchSource:0}: Error finding container 71ea15c6966b2924a5b991f5d4707a2a518f53314f73eecafa98ede1832fe5f1: Status 404 returned error can't find the container with id 71ea15c6966b2924a5b991f5d4707a2a518f53314f73eecafa98ede1832fe5f1
Nov 03 21:15:05.069415 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:15:05.069392    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb51b13_c088_4035_bd9e_91c74dfe3a95.slice/crio-e687ffeabc2401e14edd69c808b213929fd60976676b2035400d1f15b803ad1f WatchSource:0}: Error finding container e687ffeabc2401e14edd69c808b213929fd60976676b2035400d1f15b803ad1f: Status 404 returned error can't find the container with id e687ffeabc2401e14edd69c808b213929fd60976676b2035400d1f15b803ad1f
Nov 03 21:15:05.119275 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:05.119223    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-854fddcd9b-7zs44"]
Nov 03 21:15:05.121868 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:15:05.121835    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd637e4ed_6bbd_44f1_b091_4594db93e2d8.slice/crio-fff99b6497e6e45c7b14b97ce2e56930dec82e98d2b212906b78f2ca29e37ff3 WatchSource:0}: Error finding container fff99b6497e6e45c7b14b97ce2e56930dec82e98d2b212906b78f2ca29e37ff3: Status 404 returned error can't find the container with id fff99b6497e6e45c7b14b97ce2e56930dec82e98d2b212906b78f2ca29e37ff3
Nov 03 21:15:05.895187 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:05.895142    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s4x67" event={"ID":"5bb51b13-c088-4035-bd9e-91c74dfe3a95","Type":"ContainerStarted","Data":"e687ffeabc2401e14edd69c808b213929fd60976676b2035400d1f15b803ad1f"}
Nov 03 21:15:05.897099 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:05.897066    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-854fddcd9b-7zs44" event={"ID":"d637e4ed-6bbd-44f1-b091-4594db93e2d8","Type":"ContainerStarted","Data":"5aaffc9cd6025d4f6b5ac428a4a2618b7f616109b4467465bd6e09d0e4a9c972"}
Nov 03 21:15:05.897273 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:05.897109    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-854fddcd9b-7zs44" event={"ID":"d637e4ed-6bbd-44f1-b091-4594db93e2d8","Type":"ContainerStarted","Data":"fff99b6497e6e45c7b14b97ce2e56930dec82e98d2b212906b78f2ca29e37ff3"}
Nov 03 21:15:05.897354 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:05.897280    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:05.909049 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:05.909015    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-f7zn9" event={"ID":"7a77b27d-5a39-436f-b09d-21872b22709d","Type":"ContainerStarted","Data":"71ea15c6966b2924a5b991f5d4707a2a518f53314f73eecafa98ede1832fe5f1"}
Nov 03 21:15:05.909458 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:05.909430    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:15:05.911308 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:05.911289    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5ffbbbfcf7-99qbj"
Nov 03 21:15:06.022323 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:06.022255    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-854fddcd9b-7zs44" podStartSLOduration=2.022220547 podStartE2EDuration="2.022220547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:06.021345257 +0000 UTC m=+20.515267137" watchObservedRunningTime="2026-11-03 21:15:06.022220547 +0000 UTC m=+20.516142424"
Nov 03 21:15:06.828231 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:06.828196    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cccr5"
Nov 03 21:15:06.916565 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:06.916533    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s4x67" event={"ID":"5bb51b13-c088-4035-bd9e-91c74dfe3a95","Type":"ContainerStarted","Data":"e14b003d270098ab498cfdcdf9a4937ec8f2e810aa1794b8ddedc9a2f224530f"}
Nov 03 21:15:06.916950 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:06.916684    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s4x67"
Nov 03 21:15:06.922828 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:06.922774    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s4x67"
Nov 03 21:15:07.030620 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:07.030565    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s4x67" podStartSLOduration=1.432490341 podStartE2EDuration="3.030547416s" totalImagesPullingTime="1.598057075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:07.029305012 +0000 UTC m=+21.523226889" watchObservedRunningTime="2026-11-03 21:15:07.030547416 +0000 UTC m=+21.524469292"
Nov 03 21:15:07.927082 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:07.927040    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v9vxl" event={"ID":"cad3df5d-3fe9-449b-8b70-793bceb47985","Type":"ContainerStarted","Data":"f1c9eefcf872ce4564dc58565591381e47099a983d25d8b17b00cf9b41bbd867"}
Nov 03 21:15:08.160360 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:08.160321    5053 patch_prober.go:29] interesting pod/image-registry-75cb56744b-c6rsx container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 21:15:08.160514 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:08.160381    5053 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-75cb56744b-c6rsx" podUID="ef7a859e-70f5-497a-9ad4-6614885fb71e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:15:10.934155 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:15:10.934110    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:15:14.341891 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:14.341858    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:15:15.284649 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.284617    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vqgbq"]
Nov 03 21:15:15.302885 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.302854    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.360440 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.360408    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 21:15:15.360897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.360468    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 21:15:15.374717 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.374686    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bk9l\" (UniqueName: \"kubernetes.io/projected/4f856e5a-577a-45b0-92dd-61570d9a0a36-kube-api-access-9bk9l\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.374900 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.374729    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4f856e5a-577a-45b0-92dd-61570d9a0a36-node-exporter-wtmp\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.374900 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.374759    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f856e5a-577a-45b0-92dd-61570d9a0a36-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.374900 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.374812    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4f856e5a-577a-45b0-92dd-61570d9a0a36-node-exporter-textfile\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.375034 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.374899    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f856e5a-577a-45b0-92dd-61570d9a0a36-sys\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.375034 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.374931    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4f856e5a-577a-45b0-92dd-61570d9a0a36-root\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.375034 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.374950    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4f856e5a-577a-45b0-92dd-61570d9a0a36-node-exporter-tls\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.375034 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.374969    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f856e5a-577a-45b0-92dd-61570d9a0a36-metrics-client-ca\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.375183 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.374991    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4f856e5a-577a-45b0-92dd-61570d9a0a36-node-exporter-accelerators-collector-config\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.404298 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.404265    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-79p29\""
Nov 03 21:15:15.425516 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.425486    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:15:15.453943 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.453913    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 21:15:15.454122 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.454046    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 21:15:15.477770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.477101    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4f856e5a-577a-45b0-92dd-61570d9a0a36-node-exporter-wtmp\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.477770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.477188    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f856e5a-577a-45b0-92dd-61570d9a0a36-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.477770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.477226    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4f856e5a-577a-45b0-92dd-61570d9a0a36-node-exporter-textfile\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.477770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.477283    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4f856e5a-577a-45b0-92dd-61570d9a0a36-node-exporter-wtmp\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.477770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.477289    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f856e5a-577a-45b0-92dd-61570d9a0a36-sys\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.477770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.477337    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f856e5a-577a-45b0-92dd-61570d9a0a36-sys\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.477770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.477358    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4f856e5a-577a-45b0-92dd-61570d9a0a36-root\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.477770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.477387    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4f856e5a-577a-45b0-92dd-61570d9a0a36-node-exporter-tls\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.477770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.477425    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f856e5a-577a-45b0-92dd-61570d9a0a36-metrics-client-ca\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.477770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.477456    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4f856e5a-577a-45b0-92dd-61570d9a0a36-node-exporter-accelerators-collector-config\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.477770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.477511    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4f856e5a-577a-45b0-92dd-61570d9a0a36-root\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.477770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.477587    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bk9l\" (UniqueName: \"kubernetes.io/projected/4f856e5a-577a-45b0-92dd-61570d9a0a36-kube-api-access-9bk9l\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.478650 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.478227    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f856e5a-577a-45b0-92dd-61570d9a0a36-metrics-client-ca\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.478650 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.478307    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4f856e5a-577a-45b0-92dd-61570d9a0a36-node-exporter-accelerators-collector-config\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.478787 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.477721    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4f856e5a-577a-45b0-92dd-61570d9a0a36-node-exporter-textfile\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.480381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.480359    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4f856e5a-577a-45b0-92dd-61570d9a0a36-node-exporter-tls\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.480799 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.480779    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f856e5a-577a-45b0-92dd-61570d9a0a36-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.541910 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.541831    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bk9l\" (UniqueName: \"kubernetes.io/projected/4f856e5a-577a-45b0-92dd-61570d9a0a36-kube-api-access-9bk9l\") pod \"node-exporter-vqgbq\" (UID: \"4f856e5a-577a-45b0-92dd-61570d9a0a36\") " pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.658688 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.658651    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vqgbq"
Nov 03 21:15:15.699919 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:15:15.699876    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f856e5a_577a_45b0_92dd_61570d9a0a36.slice/crio-cba7187a95cd8e9952fe8301a35b99ba00852c05122d5919b27627325a5d723f WatchSource:0}: Error finding container cba7187a95cd8e9952fe8301a35b99ba00852c05122d5919b27627325a5d723f: Status 404 returned error can't find the container with id cba7187a95cd8e9952fe8301a35b99ba00852c05122d5919b27627325a5d723f
Nov 03 21:15:15.970784 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:15.970742    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vqgbq" event={"ID":"4f856e5a-577a-45b0-92dd-61570d9a0a36","Type":"ContainerStarted","Data":"cba7187a95cd8e9952fe8301a35b99ba00852c05122d5919b27627325a5d723f"}
Nov 03 21:15:17.988260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:17.988198    5053 generic.go:356] "Generic (PLEG): container finished" podID="4f856e5a-577a-45b0-92dd-61570d9a0a36" containerID="10a01db0566ecebce965dc63dae50461f89d70e78a7a31ebc53f1864983ca935" exitCode=0
Nov 03 21:15:17.988701 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:17.988281    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vqgbq" event={"ID":"4f856e5a-577a-45b0-92dd-61570d9a0a36","Type":"ContainerDied","Data":"10a01db0566ecebce965dc63dae50461f89d70e78a7a31ebc53f1864983ca935"}
Nov 03 21:15:18.996163 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:18.996120    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vqgbq" event={"ID":"4f856e5a-577a-45b0-92dd-61570d9a0a36","Type":"ContainerStarted","Data":"451d666293a137f4f7322f3eb62f153d228f3ba36023a6d89adb117ab9b30e11"}
Nov 03 21:15:18.996622 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:18.996198    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vqgbq" event={"ID":"4f856e5a-577a-45b0-92dd-61570d9a0a36","Type":"ContainerStarted","Data":"aa3fa928392d2ce2d3eb6cfedef04b55b96eede0a075e234e676bf73c9619fae"}
Nov 03 21:15:19.112894 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:19.112847    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vqgbq" podStartSLOduration=2.121140168 podStartE2EDuration="4.11283304s" totalImagesPullingTime="1.90377033s" totalInitContainerRuntime="87.922542ms" isStatefulPod=true podCreationTimestamp="2026-11-03 21:15:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:19.111309988 +0000 UTC m=+33.605231863" watchObservedRunningTime="2026-11-03 21:15:19.11283304 +0000 UTC m=+33.606754913"
Nov 03 21:15:21.273136 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:21.273105    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8w5st"
Nov 03 21:15:23.024658 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:23.024614    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-f7zn9" event={"ID":"7a77b27d-5a39-436f-b09d-21872b22709d","Type":"ContainerStarted","Data":"adfb00cb5784f9b9913780f4fac038ade0aa668ac41b59e6c9e1f9117fe2031d"}
Nov 03 21:15:23.025198 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:23.024837    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-f7zn9"
Nov 03 21:15:23.041727 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:23.041703    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-f7zn9"
Nov 03 21:15:23.238051 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:23.237990    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-f7zn9" podStartSLOduration=2.381660707 podStartE2EDuration="19.237971018s" totalImagesPullingTime="16.856310311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:23.188167845 +0000 UTC m=+37.682089722" watchObservedRunningTime="2026-11-03 21:15:23.237971018 +0000 UTC m=+37.731892895"
Nov 03 21:15:26.928558 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:26.927444    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-854fddcd9b-7zs44"
Nov 03 21:15:27.147375 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:27.147338    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75cb56744b-c6rsx"]
Nov 03 21:15:29.355391 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.355338    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6df897dfcf-2b92c" podUID="1db07cc2-aa71-490e-a21a-0a8cb06d94d9" containerName="registry" containerID="cri-o://35a3a7301caceb4f3a2db88db71fb7cf2aa32666554ee3abe83920847084c6d2" gracePeriod=30
Nov 03 21:15:29.662055 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.662023    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:15:29.814332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.814281    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls\") pod \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") "
Nov 03 21:15:29.814539 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.814342    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-kube-api-access-lp68p\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-kube-api-access-lp68p\") pod \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") "
Nov 03 21:15:29.814539 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.814459    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-certificates\") pod \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") "
Nov 03 21:15:29.814539 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.814536    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-image-registry-private-configuration\") pod \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") "
Nov 03 21:15:29.814713 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.814583    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-trusted-ca\") pod \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") "
Nov 03 21:15:29.814713 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.814642    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-installation-pull-secrets\") pod \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") "
Nov 03 21:15:29.814713 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.814673    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-bound-sa-token\") pod \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") "
Nov 03 21:15:29.814870 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.814729    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-ca-trust-extracted\") pod \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\" (UID: \"1db07cc2-aa71-490e-a21a-0a8cb06d94d9\") "
Nov 03 21:15:29.815143 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.815093    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-certificates" pod "1db07cc2-aa71-490e-a21a-0a8cb06d94d9" (UID: "1db07cc2-aa71-490e-a21a-0a8cb06d94d9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:29.815835 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.815795    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-trusted-ca" pod "1db07cc2-aa71-490e-a21a-0a8cb06d94d9" (UID: "1db07cc2-aa71-490e-a21a-0a8cb06d94d9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:29.816333 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.816308    5053 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-certificates\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:15:29.816438 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.816339    5053 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-trusted-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:15:29.818061 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.818035    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-image-registry-private-configuration" pod "1db07cc2-aa71-490e-a21a-0a8cb06d94d9" (UID: "1db07cc2-aa71-490e-a21a-0a8cb06d94d9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:15:29.818422 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.818393    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-kube-api-access-lp68p" pod "1db07cc2-aa71-490e-a21a-0a8cb06d94d9" (UID: "1db07cc2-aa71-490e-a21a-0a8cb06d94d9"). InnerVolumeSpecName "kube-api-access-lp68p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:29.818519 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.818398    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-installation-pull-secrets" pod "1db07cc2-aa71-490e-a21a-0a8cb06d94d9" (UID: "1db07cc2-aa71-490e-a21a-0a8cb06d94d9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:15:29.818913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.818889    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-bound-sa-token" pod "1db07cc2-aa71-490e-a21a-0a8cb06d94d9" (UID: "1db07cc2-aa71-490e-a21a-0a8cb06d94d9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:29.821378 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.821351    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls" pod "1db07cc2-aa71-490e-a21a-0a8cb06d94d9" (UID: "1db07cc2-aa71-490e-a21a-0a8cb06d94d9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:29.823868 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.823806    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-ca-trust-extracted" pod "1db07cc2-aa71-490e-a21a-0a8cb06d94d9" (UID: "1db07cc2-aa71-490e-a21a-0a8cb06d94d9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:29.917542 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.917460    5053 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-image-registry-private-configuration\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:15:29.917542 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.917492    5053 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-installation-pull-secrets\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:15:29.917542 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.917510    5053 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-bound-sa-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:15:29.917542 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.917524    5053 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-ca-trust-extracted\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:15:29.917542 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.917538    5053 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-registry-tls\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:15:29.917809 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:29.917551    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp68p\" (UniqueName: \"kubernetes.io/projected/1db07cc2-aa71-490e-a21a-0a8cb06d94d9-kube-api-access-lp68p\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:15:30.062015 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:30.061974    5053 generic.go:356] "Generic (PLEG): container finished" podID="1db07cc2-aa71-490e-a21a-0a8cb06d94d9" containerID="35a3a7301caceb4f3a2db88db71fb7cf2aa32666554ee3abe83920847084c6d2" exitCode=0
Nov 03 21:15:30.062228 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:30.062057    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6df897dfcf-2b92c" event={"ID":"1db07cc2-aa71-490e-a21a-0a8cb06d94d9","Type":"ContainerDied","Data":"35a3a7301caceb4f3a2db88db71fb7cf2aa32666554ee3abe83920847084c6d2"}
Nov 03 21:15:30.062228 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:30.062106    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6df897dfcf-2b92c" event={"ID":"1db07cc2-aa71-490e-a21a-0a8cb06d94d9","Type":"ContainerDied","Data":"78fdf90155ed86aec2073837c54cde87dd033c496fecac9e4619cb4177bb84fd"}
Nov 03 21:15:30.062228 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:30.062129    5053 scope.go:122] "RemoveContainer" containerID="35a3a7301caceb4f3a2db88db71fb7cf2aa32666554ee3abe83920847084c6d2"
Nov 03 21:15:30.062228 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:30.062064    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6df897dfcf-2b92c"
Nov 03 21:15:30.079367 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:30.079347    5053 scope.go:122] "RemoveContainer" containerID="35a3a7301caceb4f3a2db88db71fb7cf2aa32666554ee3abe83920847084c6d2"
Nov 03 21:15:30.079760 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:15:30.079734    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a3a7301caceb4f3a2db88db71fb7cf2aa32666554ee3abe83920847084c6d2\": container with ID starting with 35a3a7301caceb4f3a2db88db71fb7cf2aa32666554ee3abe83920847084c6d2 not found: ID does not exist" containerID="35a3a7301caceb4f3a2db88db71fb7cf2aa32666554ee3abe83920847084c6d2"
Nov 03 21:15:30.079877 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:30.079772    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a3a7301caceb4f3a2db88db71fb7cf2aa32666554ee3abe83920847084c6d2"} err="failed to get container status \"35a3a7301caceb4f3a2db88db71fb7cf2aa32666554ee3abe83920847084c6d2\": rpc error: code = NotFound desc = could not find container \"35a3a7301caceb4f3a2db88db71fb7cf2aa32666554ee3abe83920847084c6d2\": container with ID starting with 35a3a7301caceb4f3a2db88db71fb7cf2aa32666554ee3abe83920847084c6d2 not found: ID does not exist"
Nov 03 21:15:30.228315 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:30.228282    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6df897dfcf-2b92c"]
Nov 03 21:15:30.236989 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:30.236957    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6df897dfcf-2b92c"]
Nov 03 21:15:31.664253 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:31.664212    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1db07cc2-aa71-490e-a21a-0a8cb06d94d9" path="/var/lib/kubelet/pods/1db07cc2-aa71-490e-a21a-0a8cb06d94d9/volumes"
Nov 03 21:15:40.929185 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:15:40.929145    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:15:52.159582 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.159543    5053 generic.go:356] "Generic (PLEG): container finished" podID="067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa" containerID="48f1317b67a9f3effa8ab84804bda042214866a8cd3e7213d9e7ab044f795c78" exitCode=0
Nov 03 21:15:52.160069 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.159619    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8" event={"ID":"067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa","Type":"ContainerDied","Data":"48f1317b67a9f3effa8ab84804bda042214866a8cd3e7213d9e7ab044f795c78"}
Nov 03 21:15:52.160493 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.160469    5053 scope.go:122] "RemoveContainer" containerID="48f1317b67a9f3effa8ab84804bda042214866a8cd3e7213d9e7ab044f795c78"
Nov 03 21:15:52.173076 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.173045    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-75cb56744b-c6rsx" podUID="ef7a859e-70f5-497a-9ad4-6614885fb71e" containerName="registry" containerID="cri-o://d84f3f05da84fdf8843c1da4fc34c3f0153792c25ddf218ccb92a6c9b927b92a" gracePeriod=30
Nov 03 21:15:52.481961 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.481927    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:15:52.509993 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.509955    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef7a859e-70f5-497a-9ad4-6614885fb71e-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ef7a859e-70f5-497a-9ad4-6614885fb71e-ca-trust-extracted\") pod \"ef7a859e-70f5-497a-9ad4-6614885fb71e\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") "
Nov 03 21:15:52.510179 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.510008    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef7a859e-70f5-497a-9ad4-6614885fb71e-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef7a859e-70f5-497a-9ad4-6614885fb71e-trusted-ca\") pod \"ef7a859e-70f5-497a-9ad4-6614885fb71e\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") "
Nov 03 21:15:52.510179 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.510037    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-kube-api-access-bhdwv\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-kube-api-access-bhdwv\") pod \"ef7a859e-70f5-497a-9ad4-6614885fb71e\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") "
Nov 03 21:15:52.510179 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.510064    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-bound-sa-token\") pod \"ef7a859e-70f5-497a-9ad4-6614885fb71e\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") "
Nov 03 21:15:52.510179 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.510091    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-certificates\") pod \"ef7a859e-70f5-497a-9ad4-6614885fb71e\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") "
Nov 03 21:15:52.510179 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.510147    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef7a859e-70f5-497a-9ad4-6614885fb71e-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ef7a859e-70f5-497a-9ad4-6614885fb71e-image-registry-private-configuration\") pod \"ef7a859e-70f5-497a-9ad4-6614885fb71e\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") "
Nov 03 21:15:52.510466 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.510189    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef7a859e-70f5-497a-9ad4-6614885fb71e-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ef7a859e-70f5-497a-9ad4-6614885fb71e-installation-pull-secrets\") pod \"ef7a859e-70f5-497a-9ad4-6614885fb71e\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") "
Nov 03 21:15:52.510466 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.510279    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls\") pod \"ef7a859e-70f5-497a-9ad4-6614885fb71e\" (UID: \"ef7a859e-70f5-497a-9ad4-6614885fb71e\") "
Nov 03 21:15:52.510628 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.510573    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-certificates" pod "ef7a859e-70f5-497a-9ad4-6614885fb71e" (UID: "ef7a859e-70f5-497a-9ad4-6614885fb71e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:52.510628 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.510584    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7a859e-70f5-497a-9ad4-6614885fb71e-trusted-ca" pod "ef7a859e-70f5-497a-9ad4-6614885fb71e" (UID: "ef7a859e-70f5-497a-9ad4-6614885fb71e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:52.512887 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.512854    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7a859e-70f5-497a-9ad4-6614885fb71e-installation-pull-secrets" pod "ef7a859e-70f5-497a-9ad4-6614885fb71e" (UID: "ef7a859e-70f5-497a-9ad4-6614885fb71e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:15:52.513096 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.513075    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-bound-sa-token" pod "ef7a859e-70f5-497a-9ad4-6614885fb71e" (UID: "ef7a859e-70f5-497a-9ad4-6614885fb71e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:52.513225 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.513185    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-kube-api-access-bhdwv" pod "ef7a859e-70f5-497a-9ad4-6614885fb71e" (UID: "ef7a859e-70f5-497a-9ad4-6614885fb71e"). InnerVolumeSpecName "kube-api-access-bhdwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:52.513333 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.513281    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7a859e-70f5-497a-9ad4-6614885fb71e-image-registry-private-configuration" pod "ef7a859e-70f5-497a-9ad4-6614885fb71e" (UID: "ef7a859e-70f5-497a-9ad4-6614885fb71e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:15:52.513333 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.513291    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls" pod "ef7a859e-70f5-497a-9ad4-6614885fb71e" (UID: "ef7a859e-70f5-497a-9ad4-6614885fb71e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:52.519426 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.519394    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7a859e-70f5-497a-9ad4-6614885fb71e-ca-trust-extracted" pod "ef7a859e-70f5-497a-9ad4-6614885fb71e" (UID: "ef7a859e-70f5-497a-9ad4-6614885fb71e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:52.611328 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.611299    5053 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ef7a859e-70f5-497a-9ad4-6614885fb71e-image-registry-private-configuration\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:15:52.611328 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.611326    5053 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ef7a859e-70f5-497a-9ad4-6614885fb71e-installation-pull-secrets\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:15:52.611540 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.611341    5053 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-tls\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:15:52.611540 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.611350    5053 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ef7a859e-70f5-497a-9ad4-6614885fb71e-ca-trust-extracted\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:15:52.611540 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.611358    5053 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef7a859e-70f5-497a-9ad4-6614885fb71e-trusted-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:15:52.611540 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.611367    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhdwv\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-kube-api-access-bhdwv\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:15:52.611540 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.611375    5053 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef7a859e-70f5-497a-9ad4-6614885fb71e-bound-sa-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:15:52.611540 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:52.611383    5053 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ef7a859e-70f5-497a-9ad4-6614885fb71e-registry-certificates\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:15:53.165018 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:53.164981    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k65s8" event={"ID":"067bc9bb-5bc9-4ef5-8118-bd9a7d4048fa","Type":"ContainerStarted","Data":"6c3896db37b0f053f6d58a09f4a501bbba6a47390d1e9fd02ece44bd46dcf68a"}
Nov 03 21:15:53.166215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:53.166194    5053 generic.go:356] "Generic (PLEG): container finished" podID="ef7a859e-70f5-497a-9ad4-6614885fb71e" containerID="d84f3f05da84fdf8843c1da4fc34c3f0153792c25ddf218ccb92a6c9b927b92a" exitCode=0
Nov 03 21:15:53.166346 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:53.166228    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75cb56744b-c6rsx"
Nov 03 21:15:53.166346 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:53.166254    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75cb56744b-c6rsx" event={"ID":"ef7a859e-70f5-497a-9ad4-6614885fb71e","Type":"ContainerDied","Data":"d84f3f05da84fdf8843c1da4fc34c3f0153792c25ddf218ccb92a6c9b927b92a"}
Nov 03 21:15:53.166414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:53.166361    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75cb56744b-c6rsx" event={"ID":"ef7a859e-70f5-497a-9ad4-6614885fb71e","Type":"ContainerDied","Data":"9ea91149f669e1acb18d4a39de853703caee922ca58e77977727b5fa889aaeb4"}
Nov 03 21:15:53.166414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:53.166379    5053 scope.go:122] "RemoveContainer" containerID="d84f3f05da84fdf8843c1da4fc34c3f0153792c25ddf218ccb92a6c9b927b92a"
Nov 03 21:15:53.174874 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:53.174854    5053 scope.go:122] "RemoveContainer" containerID="d84f3f05da84fdf8843c1da4fc34c3f0153792c25ddf218ccb92a6c9b927b92a"
Nov 03 21:15:53.175128 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:15:53.175109    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d84f3f05da84fdf8843c1da4fc34c3f0153792c25ddf218ccb92a6c9b927b92a\": container with ID starting with d84f3f05da84fdf8843c1da4fc34c3f0153792c25ddf218ccb92a6c9b927b92a not found: ID does not exist" containerID="d84f3f05da84fdf8843c1da4fc34c3f0153792c25ddf218ccb92a6c9b927b92a"
Nov 03 21:15:53.175189 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:53.175140    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d84f3f05da84fdf8843c1da4fc34c3f0153792c25ddf218ccb92a6c9b927b92a"} err="failed to get container status \"d84f3f05da84fdf8843c1da4fc34c3f0153792c25ddf218ccb92a6c9b927b92a\": rpc error: code = NotFound desc = could not find container \"d84f3f05da84fdf8843c1da4fc34c3f0153792c25ddf218ccb92a6c9b927b92a\": container with ID starting with d84f3f05da84fdf8843c1da4fc34c3f0153792c25ddf218ccb92a6c9b927b92a not found: ID does not exist"
Nov 03 21:15:53.388223 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:53.388192    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75cb56744b-c6rsx"]
Nov 03 21:15:53.391518 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:53.391493    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-75cb56744b-c6rsx"]
Nov 03 21:15:53.664776 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:15:53.664746    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef7a859e-70f5-497a-9ad4-6614885fb71e" path="/var/lib/kubelet/pods/ef7a859e-70f5-497a-9ad4-6614885fb71e/volumes"
Nov 03 21:16:08.227121 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:08.227084    5053 generic.go:356] "Generic (PLEG): container finished" podID="caaa9f42-f8a2-4459-8917-a53e8fd852cc" containerID="6ae6898c171f21f23843d81effbaf5336ef32b69a67d3c34a5e14d14e6a3a3d2" exitCode=0
Nov 03 21:16:08.227667 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:08.227162    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-g82mg-mqwqf" event={"ID":"caaa9f42-f8a2-4459-8917-a53e8fd852cc","Type":"ContainerDied","Data":"6ae6898c171f21f23843d81effbaf5336ef32b69a67d3c34a5e14d14e6a3a3d2"}
Nov 03 21:16:09.376048 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:09.376022    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-g82mg-mqwqf"
Nov 03 21:16:09.441957 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:09.441927    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caaa9f42-f8a2-4459-8917-a53e8fd852cc-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/caaa9f42-f8a2-4459-8917-a53e8fd852cc-archives-path\") pod \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\" (UID: \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\") "
Nov 03 21:16:09.442130 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:09.441963    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caaa9f42-f8a2-4459-8917-a53e8fd852cc-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caaa9f42-f8a2-4459-8917-a53e8fd852cc-trusted-ca-bundle\") pod \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\" (UID: \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\") "
Nov 03 21:16:09.442130 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:09.441995    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caaa9f42-f8a2-4459-8917-a53e8fd852cc-kube-api-access-mddbv\" (UniqueName: \"kubernetes.io/projected/caaa9f42-f8a2-4459-8917-a53e8fd852cc-kube-api-access-mddbv\") pod \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\" (UID: \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\") "
Nov 03 21:16:09.442130 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:09.442022    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caaa9f42-f8a2-4459-8917-a53e8fd852cc-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caaa9f42-f8a2-4459-8917-a53e8fd852cc-service-ca-bundle\") pod \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\" (UID: \"caaa9f42-f8a2-4459-8917-a53e8fd852cc\") "
Nov 03 21:16:09.442379 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:09.442354    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caaa9f42-f8a2-4459-8917-a53e8fd852cc-trusted-ca-bundle" pod "caaa9f42-f8a2-4459-8917-a53e8fd852cc" (UID: "caaa9f42-f8a2-4459-8917-a53e8fd852cc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:09.442427 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:09.442360    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caaa9f42-f8a2-4459-8917-a53e8fd852cc-archives-path" pod "caaa9f42-f8a2-4459-8917-a53e8fd852cc" (UID: "caaa9f42-f8a2-4459-8917-a53e8fd852cc"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:09.442465 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:09.442427    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caaa9f42-f8a2-4459-8917-a53e8fd852cc-service-ca-bundle" pod "caaa9f42-f8a2-4459-8917-a53e8fd852cc" (UID: "caaa9f42-f8a2-4459-8917-a53e8fd852cc"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:09.444415 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:09.444392    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caaa9f42-f8a2-4459-8917-a53e8fd852cc-kube-api-access-mddbv" pod "caaa9f42-f8a2-4459-8917-a53e8fd852cc" (UID: "caaa9f42-f8a2-4459-8917-a53e8fd852cc"). InnerVolumeSpecName "kube-api-access-mddbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:09.543565 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:09.543480    5053 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/caaa9f42-f8a2-4459-8917-a53e8fd852cc-archives-path\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:16:09.543565 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:09.543509    5053 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caaa9f42-f8a2-4459-8917-a53e8fd852cc-trusted-ca-bundle\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:16:09.543565 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:09.543519    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mddbv\" (UniqueName: \"kubernetes.io/projected/caaa9f42-f8a2-4459-8917-a53e8fd852cc-kube-api-access-mddbv\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:16:09.543565 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:09.543528    5053 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/caaa9f42-f8a2-4459-8917-a53e8fd852cc-service-ca-bundle\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:16:10.236349 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:10.236309    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-g82mg-mqwqf" event={"ID":"caaa9f42-f8a2-4459-8917-a53e8fd852cc","Type":"ContainerDied","Data":"e6af7c60346a713e8a6887b47105e85cf45efcebfd0f257edcc8cdf6bff15154"}
Nov 03 21:16:10.236522 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:10.236360    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6af7c60346a713e8a6887b47105e85cf45efcebfd0f257edcc8cdf6bff15154"
Nov 03 21:16:10.236522 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:10.236325    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-g82mg-mqwqf"
Nov 03 21:16:10.928228 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:16:10.928190    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:16:16.260614 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:16.260582    5053 generic.go:356] "Generic (PLEG): container finished" podID="5cacf547-c691-4b14-936c-e6b08691bf93" containerID="5a02761cdb34a1806e2c1620bf7eff9ff6f6f8477be95f582877726937c31d50" exitCode=0
Nov 03 21:16:16.261055 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:16.260654    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-bc6rw" event={"ID":"5cacf547-c691-4b14-936c-e6b08691bf93","Type":"ContainerDied","Data":"5a02761cdb34a1806e2c1620bf7eff9ff6f6f8477be95f582877726937c31d50"}
Nov 03 21:16:16.261055 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:16.261000    5053 scope.go:122] "RemoveContainer" containerID="5a02761cdb34a1806e2c1620bf7eff9ff6f6f8477be95f582877726937c31d50"
Nov 03 21:16:17.266538 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.266505    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-bc6rw" event={"ID":"5cacf547-c691-4b14-936c-e6b08691bf93","Type":"ContainerStarted","Data":"a59f26525a7fcc2b2e99adfd4ab11295cda416af9dbdd92e283d05279252ed56"}
Nov 03 21:16:17.489527 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.489494    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-lhfb6-9jhnr"]
Nov 03 21:16:17.489829 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.489814    5053 cpu_manager.go:414] "containerMap: removing container" podUID="caaa9f42-f8a2-4459-8917-a53e8fd852cc" containerName="insights-gathering"
Nov 03 21:16:17.489870 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.489832    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caaa9f42-f8a2-4459-8917-a53e8fd852cc" containerName="insights-gathering"
Nov 03 21:16:17.489870 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.489843    5053 cpu_manager.go:414] "containerMap: removing container" podUID="1db07cc2-aa71-490e-a21a-0a8cb06d94d9" containerName="registry"
Nov 03 21:16:17.489870 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.489848    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db07cc2-aa71-490e-a21a-0a8cb06d94d9" containerName="registry"
Nov 03 21:16:17.489870 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.489864    5053 cpu_manager.go:414] "containerMap: removing container" podUID="ef7a859e-70f5-497a-9ad4-6614885fb71e" containerName="registry"
Nov 03 21:16:17.489870 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.489868    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7a859e-70f5-497a-9ad4-6614885fb71e" containerName="registry"
Nov 03 21:16:17.490014 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.489967    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="caaa9f42-f8a2-4459-8917-a53e8fd852cc" containerName="insights-gathering"
Nov 03 21:16:17.490014 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.489977    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db07cc2-aa71-490e-a21a-0a8cb06d94d9" containerName="registry"
Nov 03 21:16:17.490014 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.489990    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7a859e-70f5-497a-9ad4-6614885fb71e" containerName="registry"
Nov 03 21:16:17.491715 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.491697    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lhfb6-9jhnr"
Nov 03 21:16:17.543400 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.543341    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-md6qp\""
Nov 03 21:16:17.547795 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.547772    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-lhfb6-9jhnr"]
Nov 03 21:16:17.603960 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.603929    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-776pn\" (UniqueName: \"kubernetes.io/projected/cad4973e-e503-4c91-9c32-42380927fdb3-kube-api-access-776pn\") pod \"periodic-gathering-lhfb6-9jhnr\" (UID: \"cad4973e-e503-4c91-9c32-42380927fdb3\") " pod="openshift-insights/periodic-gathering-lhfb6-9jhnr"
Nov 03 21:16:17.604072 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.604035    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cad4973e-e503-4c91-9c32-42380927fdb3-trusted-ca-bundle\") pod \"periodic-gathering-lhfb6-9jhnr\" (UID: \"cad4973e-e503-4c91-9c32-42380927fdb3\") " pod="openshift-insights/periodic-gathering-lhfb6-9jhnr"
Nov 03 21:16:17.604116 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.604092    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cad4973e-e503-4c91-9c32-42380927fdb3-archives-path\") pod \"periodic-gathering-lhfb6-9jhnr\" (UID: \"cad4973e-e503-4c91-9c32-42380927fdb3\") " pod="openshift-insights/periodic-gathering-lhfb6-9jhnr"
Nov 03 21:16:17.604152 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.604125    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cad4973e-e503-4c91-9c32-42380927fdb3-service-ca-bundle\") pod \"periodic-gathering-lhfb6-9jhnr\" (UID: \"cad4973e-e503-4c91-9c32-42380927fdb3\") " pod="openshift-insights/periodic-gathering-lhfb6-9jhnr"
Nov 03 21:16:17.705476 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.705427    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cad4973e-e503-4c91-9c32-42380927fdb3-archives-path\") pod \"periodic-gathering-lhfb6-9jhnr\" (UID: \"cad4973e-e503-4c91-9c32-42380927fdb3\") " pod="openshift-insights/periodic-gathering-lhfb6-9jhnr"
Nov 03 21:16:17.705476 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.705478    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cad4973e-e503-4c91-9c32-42380927fdb3-service-ca-bundle\") pod \"periodic-gathering-lhfb6-9jhnr\" (UID: \"cad4973e-e503-4c91-9c32-42380927fdb3\") " pod="openshift-insights/periodic-gathering-lhfb6-9jhnr"
Nov 03 21:16:17.705722 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.705505    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-776pn\" (UniqueName: \"kubernetes.io/projected/cad4973e-e503-4c91-9c32-42380927fdb3-kube-api-access-776pn\") pod \"periodic-gathering-lhfb6-9jhnr\" (UID: \"cad4973e-e503-4c91-9c32-42380927fdb3\") " pod="openshift-insights/periodic-gathering-lhfb6-9jhnr"
Nov 03 21:16:17.705722 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.705543    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cad4973e-e503-4c91-9c32-42380927fdb3-trusted-ca-bundle\") pod \"periodic-gathering-lhfb6-9jhnr\" (UID: \"cad4973e-e503-4c91-9c32-42380927fdb3\") " pod="openshift-insights/periodic-gathering-lhfb6-9jhnr"
Nov 03 21:16:17.705901 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.705879    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cad4973e-e503-4c91-9c32-42380927fdb3-archives-path\") pod \"periodic-gathering-lhfb6-9jhnr\" (UID: \"cad4973e-e503-4c91-9c32-42380927fdb3\") " pod="openshift-insights/periodic-gathering-lhfb6-9jhnr"
Nov 03 21:16:17.706209 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.706190    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cad4973e-e503-4c91-9c32-42380927fdb3-service-ca-bundle\") pod \"periodic-gathering-lhfb6-9jhnr\" (UID: \"cad4973e-e503-4c91-9c32-42380927fdb3\") " pod="openshift-insights/periodic-gathering-lhfb6-9jhnr"
Nov 03 21:16:17.706448 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.706432    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cad4973e-e503-4c91-9c32-42380927fdb3-trusted-ca-bundle\") pod \"periodic-gathering-lhfb6-9jhnr\" (UID: \"cad4973e-e503-4c91-9c32-42380927fdb3\") " pod="openshift-insights/periodic-gathering-lhfb6-9jhnr"
Nov 03 21:16:17.762352 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.762323    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-776pn\" (UniqueName: \"kubernetes.io/projected/cad4973e-e503-4c91-9c32-42380927fdb3-kube-api-access-776pn\") pod \"periodic-gathering-lhfb6-9jhnr\" (UID: \"cad4973e-e503-4c91-9c32-42380927fdb3\") " pod="openshift-insights/periodic-gathering-lhfb6-9jhnr"
Nov 03 21:16:17.801296 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:17.801205    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lhfb6-9jhnr"
Nov 03 21:16:18.164136 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:18.164017    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-lhfb6-9jhnr"]
Nov 03 21:16:18.166735 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:16:18.166704    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad4973e_e503_4c91_9c32_42380927fdb3.slice/crio-88b284500b3d479a59e7ba7aa427d190ce84ae9a4248c5eea370a7f656a275c8 WatchSource:0}: Error finding container 88b284500b3d479a59e7ba7aa427d190ce84ae9a4248c5eea370a7f656a275c8: Status 404 returned error can't find the container with id 88b284500b3d479a59e7ba7aa427d190ce84ae9a4248c5eea370a7f656a275c8
Nov 03 21:16:18.272875 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:18.272822    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lhfb6-9jhnr" event={"ID":"cad4973e-e503-4c91-9c32-42380927fdb3","Type":"ContainerStarted","Data":"3661603ae86d0fd144286f607977244bdb662e1171b5044428d4dbbb37a88d2c"}
Nov 03 21:16:18.273260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:18.272891    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lhfb6-9jhnr" event={"ID":"cad4973e-e503-4c91-9c32-42380927fdb3","Type":"ContainerStarted","Data":"88b284500b3d479a59e7ba7aa427d190ce84ae9a4248c5eea370a7f656a275c8"}
Nov 03 21:16:18.385324 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:16:18.385279    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-lhfb6-9jhnr" podStartSLOduration=1.3852651169999999 podStartE2EDuration="1.385265117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:16:18.384116122 +0000 UTC m=+92.878037996" watchObservedRunningTime="2026-11-03 21:16:18.385265117 +0000 UTC m=+92.879186992"
Nov 03 21:16:40.926580 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:16:40.926542    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:17:10.926427 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:17:10.926383    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:17:40.926587 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:17:40.926544    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:17:43.628013 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:17:43.627933    5053 generic.go:356] "Generic (PLEG): container finished" podID="cad4973e-e503-4c91-9c32-42380927fdb3" containerID="3661603ae86d0fd144286f607977244bdb662e1171b5044428d4dbbb37a88d2c" exitCode=0
Nov 03 21:17:43.628458 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:17:43.628010    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lhfb6-9jhnr" event={"ID":"cad4973e-e503-4c91-9c32-42380927fdb3","Type":"ContainerDied","Data":"3661603ae86d0fd144286f607977244bdb662e1171b5044428d4dbbb37a88d2c"}
Nov 03 21:17:44.758320 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:17:44.758296    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lhfb6-9jhnr"
Nov 03 21:17:44.930980 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:17:44.930878    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cad4973e-e503-4c91-9c32-42380927fdb3-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cad4973e-e503-4c91-9c32-42380927fdb3-trusted-ca-bundle\") pod \"cad4973e-e503-4c91-9c32-42380927fdb3\" (UID: \"cad4973e-e503-4c91-9c32-42380927fdb3\") "
Nov 03 21:17:44.930980 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:17:44.930924    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cad4973e-e503-4c91-9c32-42380927fdb3-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cad4973e-e503-4c91-9c32-42380927fdb3-service-ca-bundle\") pod \"cad4973e-e503-4c91-9c32-42380927fdb3\" (UID: \"cad4973e-e503-4c91-9c32-42380927fdb3\") "
Nov 03 21:17:44.930980 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:17:44.930963    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad4973e-e503-4c91-9c32-42380927fdb3-kube-api-access-776pn\" (UniqueName: \"kubernetes.io/projected/cad4973e-e503-4c91-9c32-42380927fdb3-kube-api-access-776pn\") pod \"cad4973e-e503-4c91-9c32-42380927fdb3\" (UID: \"cad4973e-e503-4c91-9c32-42380927fdb3\") "
Nov 03 21:17:44.930980 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:17:44.930985    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad4973e-e503-4c91-9c32-42380927fdb3-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cad4973e-e503-4c91-9c32-42380927fdb3-archives-path\") pod \"cad4973e-e503-4c91-9c32-42380927fdb3\" (UID: \"cad4973e-e503-4c91-9c32-42380927fdb3\") "
Nov 03 21:17:44.931377 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:17:44.931348    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad4973e-e503-4c91-9c32-42380927fdb3-service-ca-bundle" pod "cad4973e-e503-4c91-9c32-42380927fdb3" (UID: "cad4973e-e503-4c91-9c32-42380927fdb3"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:17:44.931430 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:17:44.931380    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad4973e-e503-4c91-9c32-42380927fdb3-archives-path" pod "cad4973e-e503-4c91-9c32-42380927fdb3" (UID: "cad4973e-e503-4c91-9c32-42380927fdb3"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:44.931430 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:17:44.931387    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad4973e-e503-4c91-9c32-42380927fdb3-trusted-ca-bundle" pod "cad4973e-e503-4c91-9c32-42380927fdb3" (UID: "cad4973e-e503-4c91-9c32-42380927fdb3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:17:44.933281 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:17:44.933260    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad4973e-e503-4c91-9c32-42380927fdb3-kube-api-access-776pn" pod "cad4973e-e503-4c91-9c32-42380927fdb3" (UID: "cad4973e-e503-4c91-9c32-42380927fdb3"). InnerVolumeSpecName "kube-api-access-776pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:45.032509 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:17:45.032470    5053 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cad4973e-e503-4c91-9c32-42380927fdb3-trusted-ca-bundle\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:17:45.032509 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:17:45.032499    5053 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cad4973e-e503-4c91-9c32-42380927fdb3-service-ca-bundle\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:17:45.032509 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:17:45.032508    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-776pn\" (UniqueName: \"kubernetes.io/projected/cad4973e-e503-4c91-9c32-42380927fdb3-kube-api-access-776pn\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:17:45.032509 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:17:45.032518    5053 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cad4973e-e503-4c91-9c32-42380927fdb3-archives-path\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:17:45.636796 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:17:45.636766    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lhfb6-9jhnr"
Nov 03 21:17:45.636796 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:17:45.636778    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lhfb6-9jhnr" event={"ID":"cad4973e-e503-4c91-9c32-42380927fdb3","Type":"ContainerDied","Data":"88b284500b3d479a59e7ba7aa427d190ce84ae9a4248c5eea370a7f656a275c8"}
Nov 03 21:17:45.636997 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:17:45.636825    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88b284500b3d479a59e7ba7aa427d190ce84ae9a4248c5eea370a7f656a275c8"
Nov 03 21:18:10.926228 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:18:10.926188    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:18:40.625371 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.625329    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"]
Nov 03 21:18:40.625820 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.625665    5053 cpu_manager.go:414] "containerMap: removing container" podUID="cad4973e-e503-4c91-9c32-42380927fdb3" containerName="insights-gathering"
Nov 03 21:18:40.625820 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.625676    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad4973e-e503-4c91-9c32-42380927fdb3" containerName="insights-gathering"
Nov 03 21:18:40.625820 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.625747    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad4973e-e503-4c91-9c32-42380927fdb3" containerName="insights-gathering"
Nov 03 21:18:40.627509 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.627492    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.679405 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.679373    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 21:18:40.679557 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.679427    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-mc2lq\""
Nov 03 21:18:40.730282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.730227    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"]
Nov 03 21:18:40.776345 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.776314    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.776543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.776346    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.776543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.776478    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.776543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.776513    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.776543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.776529    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.776692 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.776545    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.776725 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.776684    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q95j7\" (UniqueName: \"kubernetes.io/projected/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-kube-api-access-q95j7\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.776756 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.776727    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.776756 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.776744    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.877627 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.877549    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.877627 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.877577    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.877627 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.877599    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.877627 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.877615    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.877911 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.877653    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.877911 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.877692    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.877911 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.877716    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.877911 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.877735    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.877911 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.877776    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q95j7\" (UniqueName: \"kubernetes.io/projected/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-kube-api-access-q95j7\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.878160 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.878012    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.878160 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.878122    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.878300 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.878281    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.878351 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.878332    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.878434 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.878397    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.880096 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.880071    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.880219 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.880197    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.926036 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:18:40.926006    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:18:40.931578 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.931556    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"]
Nov 03 21:18:40.933837 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.933822    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:40.939522 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.939493    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q95j7\" (UniqueName: \"kubernetes.io/projected/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-kube-api-access-q95j7\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.941854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.941832    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-blk95\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:40.996350 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:40.995870    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"]
Nov 03 21:18:41.047133 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.047101    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-dppkd\""
Nov 03 21:18:41.079392 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.079363    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5665aa62-30d5-4ced-8b5c-fedb882823cf-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.079523 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.079401    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5665aa62-30d5-4ced-8b5c-fedb882823cf-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.079523 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.079483    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5665aa62-30d5-4ced-8b5c-fedb882823cf-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.079523 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.079513    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5665aa62-30d5-4ced-8b5c-fedb882823cf-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.079639 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.079592    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5665aa62-30d5-4ced-8b5c-fedb882823cf-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.079639 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.079623    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcbjc\" (UniqueName: \"kubernetes.io/projected/5665aa62-30d5-4ced-8b5c-fedb882823cf-kube-api-access-jcbjc\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.079699 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.079641    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5665aa62-30d5-4ced-8b5c-fedb882823cf-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.079699 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.079679    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5665aa62-30d5-4ced-8b5c-fedb882823cf-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.079781 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.079713    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5665aa62-30d5-4ced-8b5c-fedb882823cf-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.180106 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.180032    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5665aa62-30d5-4ced-8b5c-fedb882823cf-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.180106 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.180071    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcbjc\" (UniqueName: \"kubernetes.io/projected/5665aa62-30d5-4ced-8b5c-fedb882823cf-kube-api-access-jcbjc\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.180106 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.180087    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5665aa62-30d5-4ced-8b5c-fedb882823cf-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.180381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.180133    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5665aa62-30d5-4ced-8b5c-fedb882823cf-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.180381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.180156    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5665aa62-30d5-4ced-8b5c-fedb882823cf-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.180381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.180181    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5665aa62-30d5-4ced-8b5c-fedb882823cf-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.180381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.180212    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5665aa62-30d5-4ced-8b5c-fedb882823cf-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.180381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.180283    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5665aa62-30d5-4ced-8b5c-fedb882823cf-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.180381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.180308    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5665aa62-30d5-4ced-8b5c-fedb882823cf-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.180645 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.180587    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5665aa62-30d5-4ced-8b5c-fedb882823cf-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.180709 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.180686    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5665aa62-30d5-4ced-8b5c-fedb882823cf-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.180757 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.180720    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5665aa62-30d5-4ced-8b5c-fedb882823cf-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.180826 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.180805    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5665aa62-30d5-4ced-8b5c-fedb882823cf-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.181014 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.180997    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5665aa62-30d5-4ced-8b5c-fedb882823cf-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.182542 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.182522    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5665aa62-30d5-4ced-8b5c-fedb882823cf-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.182761 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.182744    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5665aa62-30d5-4ced-8b5c-fedb882823cf-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.236155 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.236134    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5665aa62-30d5-4ced-8b5c-fedb882823cf-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.236288 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.236202    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcbjc\" (UniqueName: \"kubernetes.io/projected/5665aa62-30d5-4ced-8b5c-fedb882823cf-kube-api-access-jcbjc\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9\" (UID: \"5665aa62-30d5-4ced-8b5c-fedb882823cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.238070 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.238055    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:41.257263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.257215    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:41.686505 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.686482    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"]
Nov 03 21:18:41.689069 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:18:41.689042    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c6d17e_4e96_4837_8ec2_c1da7ecf30be.slice/crio-bf4bd6d51e9ebdf363022a6884412fb60a191da27a3d5a343edd0c06d7397253 WatchSource:0}: Error finding container bf4bd6d51e9ebdf363022a6884412fb60a191da27a3d5a343edd0c06d7397253: Status 404 returned error can't find the container with id bf4bd6d51e9ebdf363022a6884412fb60a191da27a3d5a343edd0c06d7397253
Nov 03 21:18:41.692815 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.692792    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"]
Nov 03 21:18:41.709882 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:18:41.709858    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5665aa62_30d5_4ced_8b5c_fedb882823cf.slice/crio-84f74cb97b712ca290f1c26a9b9e41bfa8c7d6b67b23e000fa1cb5c5546db5c2 WatchSource:0}: Error finding container 84f74cb97b712ca290f1c26a9b9e41bfa8c7d6b67b23e000fa1cb5c5546db5c2: Status 404 returned error can't find the container with id 84f74cb97b712ca290f1c26a9b9e41bfa8c7d6b67b23e000fa1cb5c5546db5c2
Nov 03 21:18:41.849109 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.849071    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9" event={"ID":"5665aa62-30d5-4ced-8b5c-fedb882823cf","Type":"ContainerStarted","Data":"84f74cb97b712ca290f1c26a9b9e41bfa8c7d6b67b23e000fa1cb5c5546db5c2"}
Nov 03 21:18:41.850129 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:41.850105    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95" event={"ID":"48c6d17e-4e96-4837-8ec2-c1da7ecf30be","Type":"ContainerStarted","Data":"bf4bd6d51e9ebdf363022a6884412fb60a191da27a3d5a343edd0c06d7397253"}
Nov 03 21:18:45.481741 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:45.481701    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:18:45.482124 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:45.481793    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:18:45.482124 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:45.481832    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:18:45.652313 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:45.652275    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:18:45.652415 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:45.652351    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:18:45.652415 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:45.652388    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:18:45.874058 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:45.873956    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9" event={"ID":"5665aa62-30d5-4ced-8b5c-fedb882823cf","Type":"ContainerStarted","Data":"2ace6fabbf8fd4be1d5932a99c713ca999ef024102081e7bfdfa55ede06031a4"}
Nov 03 21:18:45.875737 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:45.875698    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95" event={"ID":"48c6d17e-4e96-4837-8ec2-c1da7ecf30be","Type":"ContainerStarted","Data":"7d7fb3b24a8708accbea047771ca706debce91a1360335ccf32223cc806171c7"}
Nov 03 21:18:46.040313 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:46.040266    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9" podStartSLOduration=2.100320269 podStartE2EDuration="6.040251225s" totalImagesPullingTime="3.939930956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:18:46.038946629 +0000 UTC m=+240.532868504" watchObservedRunningTime="2026-11-03 21:18:46.040251225 +0000 UTC m=+240.534173168"
Nov 03 21:18:46.156181 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:46.156082    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95" podStartSLOduration=2.36623073 podStartE2EDuration="6.156066767s" totalImagesPullingTime="3.789836037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:18:46.154104297 +0000 UTC m=+240.648026171" watchObservedRunningTime="2026-11-03 21:18:46.156066767 +0000 UTC m=+240.649988640"
Nov 03 21:18:46.238924 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:46.238887    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:46.243598 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:46.243576    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:46.257855 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:46.257836    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:46.262057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:46.262037    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:46.880735 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:46.880698    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:18:46.881193 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:46.880798    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:46.881516 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:46.881498    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9"
Nov 03 21:18:46.881859 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:18:46.881844    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:19:10.927780 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:19:10.927741    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:19:15.064795 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.064753    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"]
Nov 03 21:19:15.068363 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.068341    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"
Nov 03 21:19:15.163492 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.163459    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3b9b9bb9-ab83-4ab3-9de3-acc59e05c900-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-brxc8\" (UID: \"3b9b9bb9-ab83-4ab3-9de3-acc59e05c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"
Nov 03 21:19:15.163665 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.163500    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29cfr\" (UniqueName: \"kubernetes.io/projected/3b9b9bb9-ab83-4ab3-9de3-acc59e05c900-kube-api-access-29cfr\") pod \"opendatahub-operator-controller-manager-5745664b78-brxc8\" (UID: \"3b9b9bb9-ab83-4ab3-9de3-acc59e05c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"
Nov 03 21:19:15.163665 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.163577    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b9b9bb9-ab83-4ab3-9de3-acc59e05c900-tmp\") pod \"opendatahub-operator-controller-manager-5745664b78-brxc8\" (UID: \"3b9b9bb9-ab83-4ab3-9de3-acc59e05c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"
Nov 03 21:19:15.163665 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.163615    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b9b9bb9-ab83-4ab3-9de3-acc59e05c900-webhook-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-brxc8\" (UID: \"3b9b9bb9-ab83-4ab3-9de3-acc59e05c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"
Nov 03 21:19:15.163665 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.163638    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b9b9bb9-ab83-4ab3-9de3-acc59e05c900-metrics-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-brxc8\" (UID: \"3b9b9bb9-ab83-4ab3-9de3-acc59e05c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"
Nov 03 21:19:15.163665 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.163658    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3b9b9bb9-ab83-4ab3-9de3-acc59e05c900-manifests\") pod \"opendatahub-operator-controller-manager-5745664b78-brxc8\" (UID: \"3b9b9bb9-ab83-4ab3-9de3-acc59e05c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"
Nov 03 21:19:15.193910 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.193885    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 21:19:15.194299 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.194276    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 21:19:15.194355 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.194302    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 21:19:15.194528 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.194494    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 21:19:15.194562 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.194514    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-tmvnd\""
Nov 03 21:19:15.194649 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.194635    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 21:19:15.264758 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.264730    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b9b9bb9-ab83-4ab3-9de3-acc59e05c900-tmp\") pod \"opendatahub-operator-controller-manager-5745664b78-brxc8\" (UID: \"3b9b9bb9-ab83-4ab3-9de3-acc59e05c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"
Nov 03 21:19:15.265008 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.264779    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b9b9bb9-ab83-4ab3-9de3-acc59e05c900-webhook-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-brxc8\" (UID: \"3b9b9bb9-ab83-4ab3-9de3-acc59e05c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"
Nov 03 21:19:15.265008 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.264811    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b9b9bb9-ab83-4ab3-9de3-acc59e05c900-metrics-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-brxc8\" (UID: \"3b9b9bb9-ab83-4ab3-9de3-acc59e05c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"
Nov 03 21:19:15.265008 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.264839    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3b9b9bb9-ab83-4ab3-9de3-acc59e05c900-manifests\") pod \"opendatahub-operator-controller-manager-5745664b78-brxc8\" (UID: \"3b9b9bb9-ab83-4ab3-9de3-acc59e05c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"
Nov 03 21:19:15.265008 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.264891    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3b9b9bb9-ab83-4ab3-9de3-acc59e05c900-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-brxc8\" (UID: \"3b9b9bb9-ab83-4ab3-9de3-acc59e05c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"
Nov 03 21:19:15.265008 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.264919    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29cfr\" (UniqueName: \"kubernetes.io/projected/3b9b9bb9-ab83-4ab3-9de3-acc59e05c900-kube-api-access-29cfr\") pod \"opendatahub-operator-controller-manager-5745664b78-brxc8\" (UID: \"3b9b9bb9-ab83-4ab3-9de3-acc59e05c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"
Nov 03 21:19:15.265311 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.265295    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3b9b9bb9-ab83-4ab3-9de3-acc59e05c900-manifests\") pod \"opendatahub-operator-controller-manager-5745664b78-brxc8\" (UID: \"3b9b9bb9-ab83-4ab3-9de3-acc59e05c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"
Nov 03 21:19:15.267283 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.267258    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b9b9bb9-ab83-4ab3-9de3-acc59e05c900-tmp\") pod \"opendatahub-operator-controller-manager-5745664b78-brxc8\" (UID: \"3b9b9bb9-ab83-4ab3-9de3-acc59e05c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"
Nov 03 21:19:15.267477 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.267457    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3b9b9bb9-ab83-4ab3-9de3-acc59e05c900-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-brxc8\" (UID: \"3b9b9bb9-ab83-4ab3-9de3-acc59e05c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"
Nov 03 21:19:15.267586 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.267567    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b9b9bb9-ab83-4ab3-9de3-acc59e05c900-metrics-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-brxc8\" (UID: \"3b9b9bb9-ab83-4ab3-9de3-acc59e05c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"
Nov 03 21:19:15.267640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.267623    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b9b9bb9-ab83-4ab3-9de3-acc59e05c900-webhook-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-brxc8\" (UID: \"3b9b9bb9-ab83-4ab3-9de3-acc59e05c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"
Nov 03 21:19:15.330556 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.330494    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29cfr\" (UniqueName: \"kubernetes.io/projected/3b9b9bb9-ab83-4ab3-9de3-acc59e05c900-kube-api-access-29cfr\") pod \"opendatahub-operator-controller-manager-5745664b78-brxc8\" (UID: \"3b9b9bb9-ab83-4ab3-9de3-acc59e05c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"
Nov 03 21:19:15.352179 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.352152    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"]
Nov 03 21:19:15.383384 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.383357    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"
Nov 03 21:19:15.918289 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.918265    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"]
Nov 03 21:19:15.920200 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:19:15.920173    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9b9bb9_ab83_4ab3_9de3_acc59e05c900.slice/crio-1ac7c9baefdb4daf2b1e308f033f3090debd71ca8f7a3ef0798bbb95487deab0 WatchSource:0}: Error finding container 1ac7c9baefdb4daf2b1e308f033f3090debd71ca8f7a3ef0798bbb95487deab0: Status 404 returned error can't find the container with id 1ac7c9baefdb4daf2b1e308f033f3090debd71ca8f7a3ef0798bbb95487deab0
Nov 03 21:19:15.996169 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:15.996132    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8" event={"ID":"3b9b9bb9-ab83-4ab3-9de3-acc59e05c900","Type":"ContainerStarted","Data":"1ac7c9baefdb4daf2b1e308f033f3090debd71ca8f7a3ef0798bbb95487deab0"}
Nov 03 21:19:21.019427 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:21.019390    5053 generic.go:356] "Generic (PLEG): container finished" podID="3b9b9bb9-ab83-4ab3-9de3-acc59e05c900" containerID="8f3875fc9d83a5b9756276930ddbca1d14695e329c780cbc7eb087b69a00fb61" exitCode=0
Nov 03 21:19:21.019874 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:21.019484    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8" event={"ID":"3b9b9bb9-ab83-4ab3-9de3-acc59e05c900","Type":"ContainerDied","Data":"8f3875fc9d83a5b9756276930ddbca1d14695e329c780cbc7eb087b69a00fb61"}
Nov 03 21:19:22.029527 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:22.029492    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8" event={"ID":"3b9b9bb9-ab83-4ab3-9de3-acc59e05c900","Type":"ContainerStarted","Data":"a0dd8bb66e9220f13f279ab8dd3d2abaa42a3ed3d6019422e62fa408442aa3ad"}
Nov 03 21:19:22.029993 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:22.029618    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"
Nov 03 21:19:22.304636 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:22.304584    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8" podStartSLOduration=2.918839676 podStartE2EDuration="8.304569985s" totalImagesPullingTime="4.976485009s" totalInitContainerRuntime="409.2453ms" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:22.254662873 +0000 UTC m=+276.748584747" watchObservedRunningTime="2026-11-03 21:19:22.304569985 +0000 UTC m=+276.798491859"
Nov 03 21:19:31.108926 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:31.108893    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-jnjhr"]
Nov 03 21:19:31.112091 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:31.112075    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jnjhr"
Nov 03 21:19:31.166772 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:31.166745    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-klptz\""
Nov 03 21:19:31.207610 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:31.207569    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td88x\" (UniqueName: \"kubernetes.io/projected/bcdcbe12-1bff-4947-ab42-909bd405e7b1-kube-api-access-td88x\") pod \"authorino-operator-7b7bb4bc77-jnjhr\" (UID: \"bcdcbe12-1bff-4947-ab42-909bd405e7b1\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-jnjhr"
Nov 03 21:19:31.216468 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:31.216445    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 21:19:31.216468 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:31.216473    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 21:19:31.217880 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:31.217862    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-jnjhr"]
Nov 03 21:19:31.308300 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:31.308271    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td88x\" (UniqueName: \"kubernetes.io/projected/bcdcbe12-1bff-4947-ab42-909bd405e7b1-kube-api-access-td88x\") pod \"authorino-operator-7b7bb4bc77-jnjhr\" (UID: \"bcdcbe12-1bff-4947-ab42-909bd405e7b1\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-jnjhr"
Nov 03 21:19:31.365308 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:31.365215    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td88x\" (UniqueName: \"kubernetes.io/projected/bcdcbe12-1bff-4947-ab42-909bd405e7b1-kube-api-access-td88x\") pod \"authorino-operator-7b7bb4bc77-jnjhr\" (UID: \"bcdcbe12-1bff-4947-ab42-909bd405e7b1\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-jnjhr"
Nov 03 21:19:31.423827 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:31.423801    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jnjhr"
Nov 03 21:19:31.795646 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:31.795624    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-jnjhr"]
Nov 03 21:19:31.798203 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:19:31.798177    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcdcbe12_1bff_4947_ab42_909bd405e7b1.slice/crio-39aca79f3eeca916f2da1338ca357bb3044f74af2a92ab6ae2a69232d3423dc5 WatchSource:0}: Error finding container 39aca79f3eeca916f2da1338ca357bb3044f74af2a92ab6ae2a69232d3423dc5: Status 404 returned error can't find the container with id 39aca79f3eeca916f2da1338ca357bb3044f74af2a92ab6ae2a69232d3423dc5
Nov 03 21:19:32.073051 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:32.072967    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jnjhr" event={"ID":"bcdcbe12-1bff-4947-ab42-909bd405e7b1","Type":"ContainerStarted","Data":"39aca79f3eeca916f2da1338ca357bb3044f74af2a92ab6ae2a69232d3423dc5"}
Nov 03 21:19:33.036562 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:33.036532    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-brxc8"
Nov 03 21:19:34.245526 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:34.245487    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-qrdkr"]
Nov 03 21:19:34.253894 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:34.253863    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qrdkr"
Nov 03 21:19:34.306892 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:34.306861    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-qrdkr"]
Nov 03 21:19:34.312332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:34.312303    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-6tmn8\""
Nov 03 21:19:34.333204 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:34.333179    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkw99\" (UniqueName: \"kubernetes.io/projected/757361e7-9919-408f-9377-317923c5ffa5-kube-api-access-fkw99\") pod \"limitador-operator-controller-manager-5778bb4449-qrdkr\" (UID: \"757361e7-9919-408f-9377-317923c5ffa5\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qrdkr"
Nov 03 21:19:34.434569 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:34.434535    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkw99\" (UniqueName: \"kubernetes.io/projected/757361e7-9919-408f-9377-317923c5ffa5-kube-api-access-fkw99\") pod \"limitador-operator-controller-manager-5778bb4449-qrdkr\" (UID: \"757361e7-9919-408f-9377-317923c5ffa5\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qrdkr"
Nov 03 21:19:34.495354 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:34.495315    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkw99\" (UniqueName: \"kubernetes.io/projected/757361e7-9919-408f-9377-317923c5ffa5-kube-api-access-fkw99\") pod \"limitador-operator-controller-manager-5778bb4449-qrdkr\" (UID: \"757361e7-9919-408f-9377-317923c5ffa5\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qrdkr"
Nov 03 21:19:34.569072 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:34.568947    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qrdkr"
Nov 03 21:19:34.954388 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:34.954345    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-qrdkr"]
Nov 03 21:19:34.957452 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:19:34.957426    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757361e7_9919_408f_9377_317923c5ffa5.slice/crio-bf201db1df003c04a0c021d2604d1fc51f7d73d90dc8bd5ba7ea1ee0efe4fbc7 WatchSource:0}: Error finding container bf201db1df003c04a0c021d2604d1fc51f7d73d90dc8bd5ba7ea1ee0efe4fbc7: Status 404 returned error can't find the container with id bf201db1df003c04a0c021d2604d1fc51f7d73d90dc8bd5ba7ea1ee0efe4fbc7
Nov 03 21:19:35.086310 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:35.086275    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qrdkr" event={"ID":"757361e7-9919-408f-9377-317923c5ffa5","Type":"ContainerStarted","Data":"bf201db1df003c04a0c021d2604d1fc51f7d73d90dc8bd5ba7ea1ee0efe4fbc7"}
Nov 03 21:19:37.610360 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:37.610327    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-7twr6"]
Nov 03 21:19:37.613957 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:37.613939    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7twr6"
Nov 03 21:19:37.667173 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:37.667150    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 03 21:19:37.716155 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:37.716129    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zqnwx\""
Nov 03 21:19:37.734777 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:37.734742    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-7twr6"]
Nov 03 21:19:37.764791 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:37.764758    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkxjr\" (UniqueName: \"kubernetes.io/projected/79d0e65f-9499-4fd9-8b3c-6f70bd3468f1-kube-api-access-dkxjr\") pod \"dns-operator-controller-manager-8557cbdb84-7twr6\" (UID: \"79d0e65f-9499-4fd9-8b3c-6f70bd3468f1\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7twr6"
Nov 03 21:19:37.866386 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:37.866309    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkxjr\" (UniqueName: \"kubernetes.io/projected/79d0e65f-9499-4fd9-8b3c-6f70bd3468f1-kube-api-access-dkxjr\") pod \"dns-operator-controller-manager-8557cbdb84-7twr6\" (UID: \"79d0e65f-9499-4fd9-8b3c-6f70bd3468f1\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7twr6"
Nov 03 21:19:37.923343 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:37.923315    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkxjr\" (UniqueName: \"kubernetes.io/projected/79d0e65f-9499-4fd9-8b3c-6f70bd3468f1-kube-api-access-dkxjr\") pod \"dns-operator-controller-manager-8557cbdb84-7twr6\" (UID: \"79d0e65f-9499-4fd9-8b3c-6f70bd3468f1\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7twr6"
Nov 03 21:19:37.926057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:37.926036    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7twr6"
Nov 03 21:19:38.100808 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:38.100781    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jnjhr" event={"ID":"bcdcbe12-1bff-4947-ab42-909bd405e7b1","Type":"ContainerStarted","Data":"23ac042b3439224bf8eb1d335452109535eb8341150eadb30f5e23e080519b0b"}
Nov 03 21:19:38.101047 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:38.101032    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jnjhr"
Nov 03 21:19:38.220677 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:38.217579    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jnjhr" podStartSLOduration=1.8875570480000001 podStartE2EDuration="7.217561479s" totalImagesPullingTime="5.330004431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:38.215010102 +0000 UTC m=+292.708931978" watchObservedRunningTime="2026-11-03 21:19:38.217561479 +0000 UTC m=+292.711483355"
Nov 03 21:19:38.378157 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:38.377994    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-7twr6"]
Nov 03 21:19:38.380771 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:19:38.380744    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d0e65f_9499_4fd9_8b3c_6f70bd3468f1.slice/crio-67163d5aae97fca550ac3c98089dfa4022e8478e7065e6905bd50acd296e639d WatchSource:0}: Error finding container 67163d5aae97fca550ac3c98089dfa4022e8478e7065e6905bd50acd296e639d: Status 404 returned error can't find the container with id 67163d5aae97fca550ac3c98089dfa4022e8478e7065e6905bd50acd296e639d
Nov 03 21:19:39.106965 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:39.106917    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7twr6" event={"ID":"79d0e65f-9499-4fd9-8b3c-6f70bd3468f1","Type":"ContainerStarted","Data":"67163d5aae97fca550ac3c98089dfa4022e8478e7065e6905bd50acd296e639d"}
Nov 03 21:19:40.112974 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:40.112939    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qrdkr" event={"ID":"757361e7-9919-408f-9377-317923c5ffa5","Type":"ContainerStarted","Data":"beea8eb2acf248db81fc2fbefbcce8f00e78f6e3f016db99e93a2f9db5e87a0c"}
Nov 03 21:19:40.113401 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:40.113013    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qrdkr"
Nov 03 21:19:40.231831 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:40.231776    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qrdkr" podStartSLOduration=1.780950679 podStartE2EDuration="6.231760238s" totalImagesPullingTime="4.450809559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:40.231090972 +0000 UTC m=+294.725012859" watchObservedRunningTime="2026-11-03 21:19:40.231760238 +0000 UTC m=+294.725682138"
Nov 03 21:19:40.925981 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:19:40.925940    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:19:42.123099 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:42.123067    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7twr6" event={"ID":"79d0e65f-9499-4fd9-8b3c-6f70bd3468f1","Type":"ContainerStarted","Data":"f8fcec7948e590f5d01c3dac914c48dac1d9d63f9f7334f1b7bf2381fa799b36"}
Nov 03 21:19:42.123498 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:42.123176    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7twr6"
Nov 03 21:19:42.249083 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:42.249026    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7twr6" podStartSLOduration=1.6055908250000002 podStartE2EDuration="5.24901145s" totalImagesPullingTime="3.643420625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:42.248473521 +0000 UTC m=+296.742395409" watchObservedRunningTime="2026-11-03 21:19:42.24901145 +0000 UTC m=+296.742933325"
Nov 03 21:19:42.513144 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:42.513111    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngwtq"]
Nov 03 21:19:42.516555 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:42.516539    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngwtq"
Nov 03 21:19:42.568412 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:42.568374    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-547fd\""
Nov 03 21:19:42.608891 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:42.608850    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk84s\" (UniqueName: \"kubernetes.io/projected/9e988f30-c8ac-4357-9d38-a7720a906f06-kube-api-access-mk84s\") pod \"kuadrant-operator-controller-manager-678d87766f-ngwtq\" (UID: \"9e988f30-c8ac-4357-9d38-a7720a906f06\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngwtq"
Nov 03 21:19:42.608891 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:42.608884    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e988f30-c8ac-4357-9d38-a7720a906f06-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-ngwtq\" (UID: \"9e988f30-c8ac-4357-9d38-a7720a906f06\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngwtq"
Nov 03 21:19:42.619466 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:42.619441    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngwtq"]
Nov 03 21:19:42.709914 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:42.709876    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk84s\" (UniqueName: \"kubernetes.io/projected/9e988f30-c8ac-4357-9d38-a7720a906f06-kube-api-access-mk84s\") pod \"kuadrant-operator-controller-manager-678d87766f-ngwtq\" (UID: \"9e988f30-c8ac-4357-9d38-a7720a906f06\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngwtq"
Nov 03 21:19:42.709914 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:42.709914    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e988f30-c8ac-4357-9d38-a7720a906f06-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-ngwtq\" (UID: \"9e988f30-c8ac-4357-9d38-a7720a906f06\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngwtq"
Nov 03 21:19:42.710338 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:42.710322    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e988f30-c8ac-4357-9d38-a7720a906f06-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-ngwtq\" (UID: \"9e988f30-c8ac-4357-9d38-a7720a906f06\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngwtq"
Nov 03 21:19:42.766771 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:42.766689    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk84s\" (UniqueName: \"kubernetes.io/projected/9e988f30-c8ac-4357-9d38-a7720a906f06-kube-api-access-mk84s\") pod \"kuadrant-operator-controller-manager-678d87766f-ngwtq\" (UID: \"9e988f30-c8ac-4357-9d38-a7720a906f06\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngwtq"
Nov 03 21:19:42.828562 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:42.828527    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngwtq"
Nov 03 21:19:43.250399 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:43.250361    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngwtq"]
Nov 03 21:19:43.252984 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:19:43.252957    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e988f30_c8ac_4357_9d38_a7720a906f06.slice/crio-34210a75c94b72e4e3da262dc30d58d5e3f6691f36fdfa5d0a585d17c1177681 WatchSource:0}: Error finding container 34210a75c94b72e4e3da262dc30d58d5e3f6691f36fdfa5d0a585d17c1177681: Status 404 returned error can't find the container with id 34210a75c94b72e4e3da262dc30d58d5e3f6691f36fdfa5d0a585d17c1177681
Nov 03 21:19:44.138523 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:44.138486    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngwtq" event={"ID":"9e988f30-c8ac-4357-9d38-a7720a906f06","Type":"ContainerStarted","Data":"34210a75c94b72e4e3da262dc30d58d5e3f6691f36fdfa5d0a585d17c1177681"}
Nov 03 21:19:45.675599 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:45.675570    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 21:19:45.676498 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:45.676472    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 21:19:45.684947 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:45.684925    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:19:45.685665 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:45.685645    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:19:49.109155 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:49.109123    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jnjhr"
Nov 03 21:19:49.163860 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:49.163779    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngwtq" event={"ID":"9e988f30-c8ac-4357-9d38-a7720a906f06","Type":"ContainerStarted","Data":"2fcd78892e27c7de27fd7a800c7a7acbc66da158697446c8940f4970057cd858"}
Nov 03 21:19:49.163860 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:49.163832    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngwtq"
Nov 03 21:19:49.382695 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:49.382640    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngwtq" podStartSLOduration=1.729059168 podStartE2EDuration="7.3826247s" totalImagesPullingTime="5.653565532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:49.381529735 +0000 UTC m=+303.875451610" watchObservedRunningTime="2026-11-03 21:19:49.3826247 +0000 UTC m=+303.876546574"
Nov 03 21:19:51.118992 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:51.118960    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qrdkr"
Nov 03 21:19:52.958910 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:52.958877    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"]
Nov 03 21:19:52.962512 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:52.962495    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.014773 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.014726    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-gfc2p\""
Nov 03 21:19:53.065478 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.065440    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"]
Nov 03 21:19:53.116259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.115812    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8886b8b-7d2c-4b60-8843-651e85091077-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.116259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.115875    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8886b8b-7d2c-4b60-8843-651e85091077-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.116259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.115911    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8886b8b-7d2c-4b60-8843-651e85091077-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.116259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.115937    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd9s9\" (UniqueName: \"kubernetes.io/projected/e8886b8b-7d2c-4b60-8843-651e85091077-kube-api-access-sd9s9\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.116259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.115999    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8886b8b-7d2c-4b60-8843-651e85091077-istio-data\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.116259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.116030    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8886b8b-7d2c-4b60-8843-651e85091077-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.116259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.116067    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8886b8b-7d2c-4b60-8843-651e85091077-istio-token\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.116259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.116098    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8886b8b-7d2c-4b60-8843-651e85091077-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.116259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.116149    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8886b8b-7d2c-4b60-8843-651e85091077-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.130536 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.130505    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7twr6"
Nov 03 21:19:53.216794 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.216687    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8886b8b-7d2c-4b60-8843-651e85091077-istio-token\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.216794 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.216735    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8886b8b-7d2c-4b60-8843-651e85091077-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.216794 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.216773    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8886b8b-7d2c-4b60-8843-651e85091077-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.217061 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.216798    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8886b8b-7d2c-4b60-8843-651e85091077-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.217061 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.216832    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8886b8b-7d2c-4b60-8843-651e85091077-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.217061 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.217051    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8886b8b-7d2c-4b60-8843-651e85091077-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.217215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.217083    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd9s9\" (UniqueName: \"kubernetes.io/projected/e8886b8b-7d2c-4b60-8843-651e85091077-kube-api-access-sd9s9\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.217215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.217154    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8886b8b-7d2c-4b60-8843-651e85091077-istio-data\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.217215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.217188    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8886b8b-7d2c-4b60-8843-651e85091077-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.217215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.217208    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8886b8b-7d2c-4b60-8843-651e85091077-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.217466 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.217331    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8886b8b-7d2c-4b60-8843-651e85091077-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.217466 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.217378    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8886b8b-7d2c-4b60-8843-651e85091077-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.217543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.217514    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8886b8b-7d2c-4b60-8843-651e85091077-istio-data\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.217719 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.217700    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8886b8b-7d2c-4b60-8843-651e85091077-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.219205 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.219180    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8886b8b-7d2c-4b60-8843-651e85091077-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.219627 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.219609    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8886b8b-7d2c-4b60-8843-651e85091077-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.274055 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.274019    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8886b8b-7d2c-4b60-8843-651e85091077-istio-token\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.274206 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.274142    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd9s9\" (UniqueName: \"kubernetes.io/projected/e8886b8b-7d2c-4b60-8843-651e85091077-kube-api-access-sd9s9\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485fcxmh\" (UID: \"e8886b8b-7d2c-4b60-8843-651e85091077\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.275842 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.275817    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:53.713385 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:19:53.713338    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8886b8b_7d2c_4b60_8843_651e85091077.slice/crio-5d07c1ce21492490347608b0ca2302505780ee096f2dd1e26222cdc532eef3b3 WatchSource:0}: Error finding container 5d07c1ce21492490347608b0ca2302505780ee096f2dd1e26222cdc532eef3b3: Status 404 returned error can't find the container with id 5d07c1ce21492490347608b0ca2302505780ee096f2dd1e26222cdc532eef3b3
Nov 03 21:19:53.715892 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.715854    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:19:53.715985 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.715952    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:19:53.716022 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.716000    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:19:53.727720 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:53.727690    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"]
Nov 03 21:19:54.187230 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:54.187138    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh" event={"ID":"e8886b8b-7d2c-4b60-8843-651e85091077","Type":"ContainerStarted","Data":"7f85d089cb54e6655047e21f28a47b268f755a3dcf988f7b4fb682d285622192"}
Nov 03 21:19:54.187230 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:54.187185    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh" event={"ID":"e8886b8b-7d2c-4b60-8843-651e85091077","Type":"ContainerStarted","Data":"5d07c1ce21492490347608b0ca2302505780ee096f2dd1e26222cdc532eef3b3"}
Nov 03 21:19:54.276489 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:54.276453    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:54.281542 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:54.281513    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:54.308283 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:54.308220    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh" podStartSLOduration=2.308206043 podStartE2EDuration="2.308206043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:54.306702246 +0000 UTC m=+308.800624121" watchObservedRunningTime="2026-11-03 21:19:54.308206043 +0000 UTC m=+308.802127917"
Nov 03 21:19:55.191573 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:55.191544    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:19:55.192630 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:19:55.192607    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485fcxmh"
Nov 03 21:20:00.171185 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:00.171153    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngwtq"
Nov 03 21:20:10.926553 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:20:10.926518    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:20:19.740692 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:19.740658    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-kmgzq"]
Nov 03 21:20:19.747856 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:19.747835    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kmgzq"
Nov 03 21:20:19.799768 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:19.799742    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 21:20:19.849306 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:19.849280    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 21:20:19.849565 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:19.849544    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1028ce52-b2d5-4d38-a449-b3396a58d4db-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-kmgzq\" (UID: \"1028ce52-b2d5-4d38-a449-b3396a58d4db\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kmgzq"
Nov 03 21:20:19.849651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:19.849579    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1028ce52-b2d5-4d38-a449-b3396a58d4db-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-kmgzq\" (UID: \"1028ce52-b2d5-4d38-a449-b3396a58d4db\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kmgzq"
Nov 03 21:20:19.849651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:19.849604    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xvff\" (UniqueName: \"kubernetes.io/projected/1028ce52-b2d5-4d38-a449-b3396a58d4db-kube-api-access-9xvff\") pod \"kserve-module-controller-manager-7566cf5fc8-kmgzq\" (UID: \"1028ce52-b2d5-4d38-a449-b3396a58d4db\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kmgzq"
Nov 03 21:20:19.849738 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:19.849686    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1028ce52-b2d5-4d38-a449-b3396a58d4db-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-kmgzq\" (UID: \"1028ce52-b2d5-4d38-a449-b3396a58d4db\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kmgzq"
Nov 03 21:20:19.899281 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:19.899252    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-gbqqf\""
Nov 03 21:20:19.949395 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:19.949362    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 03 21:20:19.950791 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:19.950770    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1028ce52-b2d5-4d38-a449-b3396a58d4db-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-kmgzq\" (UID: \"1028ce52-b2d5-4d38-a449-b3396a58d4db\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kmgzq"
Nov 03 21:20:19.950874 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:19.950859    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1028ce52-b2d5-4d38-a449-b3396a58d4db-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-kmgzq\" (UID: \"1028ce52-b2d5-4d38-a449-b3396a58d4db\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kmgzq"
Nov 03 21:20:19.950920 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:19.950900    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1028ce52-b2d5-4d38-a449-b3396a58d4db-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-kmgzq\" (UID: \"1028ce52-b2d5-4d38-a449-b3396a58d4db\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kmgzq"
Nov 03 21:20:19.950970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:19.950936    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xvff\" (UniqueName: \"kubernetes.io/projected/1028ce52-b2d5-4d38-a449-b3396a58d4db-kube-api-access-9xvff\") pod \"kserve-module-controller-manager-7566cf5fc8-kmgzq\" (UID: \"1028ce52-b2d5-4d38-a449-b3396a58d4db\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kmgzq"
Nov 03 21:20:19.951162 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:19.951119    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-kmgzq"]
Nov 03 21:20:19.951302 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:19.951215    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1028ce52-b2d5-4d38-a449-b3396a58d4db-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-kmgzq\" (UID: \"1028ce52-b2d5-4d38-a449-b3396a58d4db\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kmgzq"
Nov 03 21:20:19.951374 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:19.951318    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1028ce52-b2d5-4d38-a449-b3396a58d4db-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-kmgzq\" (UID: \"1028ce52-b2d5-4d38-a449-b3396a58d4db\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kmgzq"
Nov 03 21:20:19.951758 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:19.951740    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1028ce52-b2d5-4d38-a449-b3396a58d4db-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-kmgzq\" (UID: \"1028ce52-b2d5-4d38-a449-b3396a58d4db\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kmgzq"
Nov 03 21:20:20.007864 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:20.007790    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xvff\" (UniqueName: \"kubernetes.io/projected/1028ce52-b2d5-4d38-a449-b3396a58d4db-kube-api-access-9xvff\") pod \"kserve-module-controller-manager-7566cf5fc8-kmgzq\" (UID: \"1028ce52-b2d5-4d38-a449-b3396a58d4db\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kmgzq"
Nov 03 21:20:20.061576 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:20.061539    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kmgzq"
Nov 03 21:20:20.483782 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:20.483756    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-kmgzq"]
Nov 03 21:20:20.486327 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:20:20.486295    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1028ce52_b2d5_4d38_a449_b3396a58d4db.slice/crio-dfca163bc09dd4041224dd35470615da2c94f7e598325023c0d1ee9d3fe48552 WatchSource:0}: Error finding container dfca163bc09dd4041224dd35470615da2c94f7e598325023c0d1ee9d3fe48552: Status 404 returned error can't find the container with id dfca163bc09dd4041224dd35470615da2c94f7e598325023c0d1ee9d3fe48552
Nov 03 21:20:20.488230 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:20.488208    5053 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:20:21.321666 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:21.321628    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kmgzq" event={"ID":"1028ce52-b2d5-4d38-a449-b3396a58d4db","Type":"ContainerStarted","Data":"dfca163bc09dd4041224dd35470615da2c94f7e598325023c0d1ee9d3fe48552"}
Nov 03 21:20:24.338497 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:24.338463    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kmgzq" event={"ID":"1028ce52-b2d5-4d38-a449-b3396a58d4db","Type":"ContainerStarted","Data":"c08543160379b41f9a37e12d227d1a6c78cbc1cbe62f61c1ba511b5603d7a517"}
Nov 03 21:20:24.338874 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:24.338529    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kmgzq"
Nov 03 21:20:24.455375 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:24.455325    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kmgzq" podStartSLOduration=2.559418145 podStartE2EDuration="5.455308273s" totalImagesPullingTime="2.895890128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:20:24.454230022 +0000 UTC m=+338.948151891" watchObservedRunningTime="2026-11-03 21:20:24.455308273 +0000 UTC m=+338.949230149"
Nov 03 21:20:40.926929 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:20:40.926892    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:20:45.344643 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:20:45.344607    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kmgzq"
Nov 03 21:21:01.548497 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:01.548460    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pt78x"]
Nov 03 21:21:01.551698 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:01.551675    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pt78x"
Nov 03 21:21:01.608921 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:01.608886    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pt78x"]
Nov 03 21:21:01.623135 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:01.623102    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d27pb\" (UniqueName: \"kubernetes.io/projected/01bb8d54-6cba-4a24-ac55-4e15a54b87db-kube-api-access-d27pb\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pt78x\" (UID: \"01bb8d54-6cba-4a24-ac55-4e15a54b87db\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pt78x"
Nov 03 21:21:01.623337 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:01.623166    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/01bb8d54-6cba-4a24-ac55-4e15a54b87db-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pt78x\" (UID: \"01bb8d54-6cba-4a24-ac55-4e15a54b87db\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pt78x"
Nov 03 21:21:01.623399 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:01.623372    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/01bb8d54-6cba-4a24-ac55-4e15a54b87db-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pt78x\" (UID: \"01bb8d54-6cba-4a24-ac55-4e15a54b87db\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pt78x"
Nov 03 21:21:01.623438 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:01.623426    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01bb8d54-6cba-4a24-ac55-4e15a54b87db-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pt78x\" (UID: \"01bb8d54-6cba-4a24-ac55-4e15a54b87db\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pt78x"
Nov 03 21:21:01.656399 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:01.656365    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 03 21:21:01.656399 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:01.656378    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 03 21:21:01.656643 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:01.656420    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 03 21:21:01.656643 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:01.656537    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-qtrq7\""
Nov 03 21:21:01.724839 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:01.724795    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/01bb8d54-6cba-4a24-ac55-4e15a54b87db-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pt78x\" (UID: \"01bb8d54-6cba-4a24-ac55-4e15a54b87db\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pt78x"
Nov 03 21:21:01.725046 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:01.724850    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01bb8d54-6cba-4a24-ac55-4e15a54b87db-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pt78x\" (UID: \"01bb8d54-6cba-4a24-ac55-4e15a54b87db\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pt78x"
Nov 03 21:21:01.725046 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:21:01.724927    5053 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Nov 03 21:21:01.725046 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:01.724940    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d27pb\" (UniqueName: \"kubernetes.io/projected/01bb8d54-6cba-4a24-ac55-4e15a54b87db-kube-api-access-d27pb\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pt78x\" (UID: \"01bb8d54-6cba-4a24-ac55-4e15a54b87db\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pt78x"
Nov 03 21:21:01.725046 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:21:01.724996    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/01bb8d54-6cba-4a24-ac55-4e15a54b87db-proxy-source podName:01bb8d54-6cba-4a24-ac55-4e15a54b87db nodeName:}" failed. No retries permitted until 2026-11-03 21:21:02.224976219 +0000 UTC m=+376.718898094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/01bb8d54-6cba-4a24-ac55-4e15a54b87db-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-pt78x" (UID: "01bb8d54-6cba-4a24-ac55-4e15a54b87db") : configmap "usage-tenancy-proxy-source" not found
Nov 03 21:21:01.725046 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:01.725034    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/01bb8d54-6cba-4a24-ac55-4e15a54b87db-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pt78x\" (UID: \"01bb8d54-6cba-4a24-ac55-4e15a54b87db\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pt78x"
Nov 03 21:21:01.725448 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:21:01.725148    5053 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Nov 03 21:21:01.725448 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:21:01.725191    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01bb8d54-6cba-4a24-ac55-4e15a54b87db-tls-cert podName:01bb8d54-6cba-4a24-ac55-4e15a54b87db nodeName:}" failed. No retries permitted until 2026-11-03 21:21:02.225176434 +0000 UTC m=+376.719098304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/01bb8d54-6cba-4a24-ac55-4e15a54b87db-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-pt78x" (UID: "01bb8d54-6cba-4a24-ac55-4e15a54b87db") : secret "usage-logs-tenancy-proxy-tls" not found
Nov 03 21:21:01.725645 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:01.725625    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01bb8d54-6cba-4a24-ac55-4e15a54b87db-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pt78x\" (UID: \"01bb8d54-6cba-4a24-ac55-4e15a54b87db\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pt78x"
Nov 03 21:21:01.782693 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:01.782664    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d27pb\" (UniqueName: \"kubernetes.io/projected/01bb8d54-6cba-4a24-ac55-4e15a54b87db-kube-api-access-d27pb\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pt78x\" (UID: \"01bb8d54-6cba-4a24-ac55-4e15a54b87db\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pt78x"
Nov 03 21:21:02.231170 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:02.231132    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/01bb8d54-6cba-4a24-ac55-4e15a54b87db-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pt78x\" (UID: \"01bb8d54-6cba-4a24-ac55-4e15a54b87db\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pt78x"
Nov 03 21:21:02.231362 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:02.231223    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/01bb8d54-6cba-4a24-ac55-4e15a54b87db-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pt78x\" (UID: \"01bb8d54-6cba-4a24-ac55-4e15a54b87db\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pt78x"
Nov 03 21:21:02.232076 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:02.232039    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/01bb8d54-6cba-4a24-ac55-4e15a54b87db-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pt78x\" (UID: \"01bb8d54-6cba-4a24-ac55-4e15a54b87db\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pt78x"
Nov 03 21:21:02.233709 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:02.233689    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/01bb8d54-6cba-4a24-ac55-4e15a54b87db-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pt78x\" (UID: \"01bb8d54-6cba-4a24-ac55-4e15a54b87db\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pt78x"
Nov 03 21:21:02.464086 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:02.464045    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pt78x"
Nov 03 21:21:02.836464 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:02.836386    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pt78x"]
Nov 03 21:21:02.839520 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:21:02.839488    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01bb8d54_6cba_4a24_ac55_4e15a54b87db.slice/crio-598ff0034a58f722ffc4d73a908387f8ba7ce446f49ce13dd83236f72962bc6c WatchSource:0}: Error finding container 598ff0034a58f722ffc4d73a908387f8ba7ce446f49ce13dd83236f72962bc6c: Status 404 returned error can't find the container with id 598ff0034a58f722ffc4d73a908387f8ba7ce446f49ce13dd83236f72962bc6c
Nov 03 21:21:03.528513 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:03.528473    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pt78x" event={"ID":"01bb8d54-6cba-4a24-ac55-4e15a54b87db","Type":"ContainerStarted","Data":"598ff0034a58f722ffc4d73a908387f8ba7ce446f49ce13dd83236f72962bc6c"}
Nov 03 21:21:10.934690 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:21:10.934649    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:21:15.598790 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:15.598747    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pt78x" event={"ID":"01bb8d54-6cba-4a24-ac55-4e15a54b87db","Type":"ContainerStarted","Data":"01a5bcc01cded0e21670a72648cd6f13eb46f1cb4ad115d1a5a27402f87a4148"}
Nov 03 21:21:15.599161 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:15.598876    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pt78x"
Nov 03 21:21:15.714264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:15.714115    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pt78x" podStartSLOduration=2.100330721 podStartE2EDuration="14.714102901s" totalImagesPullingTime="12.61377218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:21:15.713408022 +0000 UTC m=+390.207329899" watchObservedRunningTime="2026-11-03 21:21:15.714102901 +0000 UTC m=+390.208024808"
Nov 03 21:21:26.609011 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:26.608935    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pt78x"
Nov 03 21:21:26.699215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:26.699183    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-vmp59"]
Nov 03 21:21:26.702493 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:26.702473    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-vmp59"
Nov 03 21:21:26.754705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:26.754679    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 03 21:21:26.754894 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:26.754776    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wnmmf\""
Nov 03 21:21:26.785537 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:26.785501    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-vmp59"]
Nov 03 21:21:26.866208 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:26.866128    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36ec7139-4665-4a3c-ab8e-5ac33799ca58-cert\") pod \"kserve-controller-manager-776cc66d47-vmp59\" (UID: \"36ec7139-4665-4a3c-ab8e-5ac33799ca58\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vmp59"
Nov 03 21:21:26.866208 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:26.866186    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9fjv\" (UniqueName: \"kubernetes.io/projected/36ec7139-4665-4a3c-ab8e-5ac33799ca58-kube-api-access-k9fjv\") pod \"kserve-controller-manager-776cc66d47-vmp59\" (UID: \"36ec7139-4665-4a3c-ab8e-5ac33799ca58\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vmp59"
Nov 03 21:21:26.927024 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:26.926994    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-z5jk4"]
Nov 03 21:21:26.930392 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:26.930376    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-z5jk4"
Nov 03 21:21:26.967389 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:26.967351    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36ec7139-4665-4a3c-ab8e-5ac33799ca58-cert\") pod \"kserve-controller-manager-776cc66d47-vmp59\" (UID: \"36ec7139-4665-4a3c-ab8e-5ac33799ca58\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vmp59"
Nov 03 21:21:26.967544 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:26.967408    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9fjv\" (UniqueName: \"kubernetes.io/projected/36ec7139-4665-4a3c-ab8e-5ac33799ca58-kube-api-access-k9fjv\") pod \"kserve-controller-manager-776cc66d47-vmp59\" (UID: \"36ec7139-4665-4a3c-ab8e-5ac33799ca58\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vmp59"
Nov 03 21:21:26.969807 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:26.969787    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36ec7139-4665-4a3c-ab8e-5ac33799ca58-cert\") pod \"kserve-controller-manager-776cc66d47-vmp59\" (UID: \"36ec7139-4665-4a3c-ab8e-5ac33799ca58\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vmp59"
Nov 03 21:21:26.987565 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:26.986432    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 03 21:21:26.987565 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:26.986550    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-rfkg4\""
Nov 03 21:21:26.990159 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:26.989999    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-z5jk4"]
Nov 03 21:21:27.023624 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.023599    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9fjv\" (UniqueName: \"kubernetes.io/projected/36ec7139-4665-4a3c-ab8e-5ac33799ca58-kube-api-access-k9fjv\") pod \"kserve-controller-manager-776cc66d47-vmp59\" (UID: \"36ec7139-4665-4a3c-ab8e-5ac33799ca58\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vmp59"
Nov 03 21:21:27.038841 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.038812    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-h2zs5"]
Nov 03 21:21:27.042448 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.042430    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-h2zs5"
Nov 03 21:21:27.068065 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.068040    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj6pm\" (UniqueName: \"kubernetes.io/projected/a3134aab-ba99-4e98-974a-dc26c5966860-kube-api-access-bj6pm\") pod \"model-serving-api-75bd89fdfc-z5jk4\" (UID: \"a3134aab-ba99-4e98-974a-dc26c5966860\") " pod="opendatahub/model-serving-api-75bd89fdfc-z5jk4"
Nov 03 21:21:27.068194 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.068082    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3134aab-ba99-4e98-974a-dc26c5966860-tls-certs\") pod \"model-serving-api-75bd89fdfc-z5jk4\" (UID: \"a3134aab-ba99-4e98-974a-dc26c5966860\") " pod="opendatahub/model-serving-api-75bd89fdfc-z5jk4"
Nov 03 21:21:27.096659 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.096636    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-jkmqw\""
Nov 03 21:21:27.096871 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.096729    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 03 21:21:27.096995 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.096928    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 03 21:21:27.098808 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.098787    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-h2zs5"]
Nov 03 21:21:27.169504 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.169420    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53dadcdb-7f9d-4dd1-acb8-9321d2bf3ef3-cert\") pod \"odh-model-controller-769875db56-h2zs5\" (UID: \"53dadcdb-7f9d-4dd1-acb8-9321d2bf3ef3\") " pod="opendatahub/odh-model-controller-769875db56-h2zs5"
Nov 03 21:21:27.169504 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.169468    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj6pm\" (UniqueName: \"kubernetes.io/projected/a3134aab-ba99-4e98-974a-dc26c5966860-kube-api-access-bj6pm\") pod \"model-serving-api-75bd89fdfc-z5jk4\" (UID: \"a3134aab-ba99-4e98-974a-dc26c5966860\") " pod="opendatahub/model-serving-api-75bd89fdfc-z5jk4"
Nov 03 21:21:27.169504 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.169485    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53dadcdb-7f9d-4dd1-acb8-9321d2bf3ef3-metrics-certs\") pod \"odh-model-controller-769875db56-h2zs5\" (UID: \"53dadcdb-7f9d-4dd1-acb8-9321d2bf3ef3\") " pod="opendatahub/odh-model-controller-769875db56-h2zs5"
Nov 03 21:21:27.169770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.169552    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3134aab-ba99-4e98-974a-dc26c5966860-tls-certs\") pod \"model-serving-api-75bd89fdfc-z5jk4\" (UID: \"a3134aab-ba99-4e98-974a-dc26c5966860\") " pod="opendatahub/model-serving-api-75bd89fdfc-z5jk4"
Nov 03 21:21:27.169770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.169568    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p44k7\" (UniqueName: \"kubernetes.io/projected/53dadcdb-7f9d-4dd1-acb8-9321d2bf3ef3-kube-api-access-p44k7\") pod \"odh-model-controller-769875db56-h2zs5\" (UID: \"53dadcdb-7f9d-4dd1-acb8-9321d2bf3ef3\") " pod="opendatahub/odh-model-controller-769875db56-h2zs5"
Nov 03 21:21:27.172126 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.172101    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3134aab-ba99-4e98-974a-dc26c5966860-tls-certs\") pod \"model-serving-api-75bd89fdfc-z5jk4\" (UID: \"a3134aab-ba99-4e98-974a-dc26c5966860\") " pod="opendatahub/model-serving-api-75bd89fdfc-z5jk4"
Nov 03 21:21:27.229699 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.229669    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj6pm\" (UniqueName: \"kubernetes.io/projected/a3134aab-ba99-4e98-974a-dc26c5966860-kube-api-access-bj6pm\") pod \"model-serving-api-75bd89fdfc-z5jk4\" (UID: \"a3134aab-ba99-4e98-974a-dc26c5966860\") " pod="opendatahub/model-serving-api-75bd89fdfc-z5jk4"
Nov 03 21:21:27.242434 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.242414    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-z5jk4"
Nov 03 21:21:27.270720 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.270694    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53dadcdb-7f9d-4dd1-acb8-9321d2bf3ef3-cert\") pod \"odh-model-controller-769875db56-h2zs5\" (UID: \"53dadcdb-7f9d-4dd1-acb8-9321d2bf3ef3\") " pod="opendatahub/odh-model-controller-769875db56-h2zs5"
Nov 03 21:21:27.270852 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.270738    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53dadcdb-7f9d-4dd1-acb8-9321d2bf3ef3-metrics-certs\") pod \"odh-model-controller-769875db56-h2zs5\" (UID: \"53dadcdb-7f9d-4dd1-acb8-9321d2bf3ef3\") " pod="opendatahub/odh-model-controller-769875db56-h2zs5"
Nov 03 21:21:27.270852 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.270793    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p44k7\" (UniqueName: \"kubernetes.io/projected/53dadcdb-7f9d-4dd1-acb8-9321d2bf3ef3-kube-api-access-p44k7\") pod \"odh-model-controller-769875db56-h2zs5\" (UID: \"53dadcdb-7f9d-4dd1-acb8-9321d2bf3ef3\") " pod="opendatahub/odh-model-controller-769875db56-h2zs5"
Nov 03 21:21:27.273190 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.273163    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53dadcdb-7f9d-4dd1-acb8-9321d2bf3ef3-cert\") pod \"odh-model-controller-769875db56-h2zs5\" (UID: \"53dadcdb-7f9d-4dd1-acb8-9321d2bf3ef3\") " pod="opendatahub/odh-model-controller-769875db56-h2zs5"
Nov 03 21:21:27.273330 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.273196    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53dadcdb-7f9d-4dd1-acb8-9321d2bf3ef3-metrics-certs\") pod \"odh-model-controller-769875db56-h2zs5\" (UID: \"53dadcdb-7f9d-4dd1-acb8-9321d2bf3ef3\") " pod="opendatahub/odh-model-controller-769875db56-h2zs5"
Nov 03 21:21:27.315032 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.314992    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-vmp59"
Nov 03 21:21:27.328543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.328517    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p44k7\" (UniqueName: \"kubernetes.io/projected/53dadcdb-7f9d-4dd1-acb8-9321d2bf3ef3-kube-api-access-p44k7\") pod \"odh-model-controller-769875db56-h2zs5\" (UID: \"53dadcdb-7f9d-4dd1-acb8-9321d2bf3ef3\") " pod="opendatahub/odh-model-controller-769875db56-h2zs5"
Nov 03 21:21:27.356461 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.356429    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-h2zs5"
Nov 03 21:21:27.611423 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.611399    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-z5jk4"]
Nov 03 21:21:27.613221 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:21:27.613194    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3134aab_ba99_4e98_974a_dc26c5966860.slice/crio-92a67ef508ee548894bad2e5bd32f4be0495f5ddbb29070ede0b0aa874cc4c8b WatchSource:0}: Error finding container 92a67ef508ee548894bad2e5bd32f4be0495f5ddbb29070ede0b0aa874cc4c8b: Status 404 returned error can't find the container with id 92a67ef508ee548894bad2e5bd32f4be0495f5ddbb29070ede0b0aa874cc4c8b
Nov 03 21:21:27.656997 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.656959    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-z5jk4" event={"ID":"a3134aab-ba99-4e98-974a-dc26c5966860","Type":"ContainerStarted","Data":"92a67ef508ee548894bad2e5bd32f4be0495f5ddbb29070ede0b0aa874cc4c8b"}
Nov 03 21:21:27.710948 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.710923    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-vmp59"]
Nov 03 21:21:27.712641 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:21:27.712614    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ec7139_4665_4a3c_ab8e_5ac33799ca58.slice/crio-11c17625dfbac840dd3c4f1d928b27bedd0a76cb919a48003f0198514f43b9b0 WatchSource:0}: Error finding container 11c17625dfbac840dd3c4f1d928b27bedd0a76cb919a48003f0198514f43b9b0: Status 404 returned error can't find the container with id 11c17625dfbac840dd3c4f1d928b27bedd0a76cb919a48003f0198514f43b9b0
Nov 03 21:21:27.730722 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:27.730698    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-h2zs5"]
Nov 03 21:21:27.733749 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:21:27.733720    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53dadcdb_7f9d_4dd1_acb8_9321d2bf3ef3.slice/crio-8bfd4a5e309553ba27f9cf90ee3c6b49a40a052a1e0c6ffaf455b965e1b6c80a WatchSource:0}: Error finding container 8bfd4a5e309553ba27f9cf90ee3c6b49a40a052a1e0c6ffaf455b965e1b6c80a: Status 404 returned error can't find the container with id 8bfd4a5e309553ba27f9cf90ee3c6b49a40a052a1e0c6ffaf455b965e1b6c80a
Nov 03 21:21:28.665447 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:28.665401    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-vmp59" event={"ID":"36ec7139-4665-4a3c-ab8e-5ac33799ca58","Type":"ContainerStarted","Data":"11c17625dfbac840dd3c4f1d928b27bedd0a76cb919a48003f0198514f43b9b0"}
Nov 03 21:21:28.667026 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:28.666995    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-h2zs5" event={"ID":"53dadcdb-7f9d-4dd1-acb8-9321d2bf3ef3","Type":"ContainerStarted","Data":"8bfd4a5e309553ba27f9cf90ee3c6b49a40a052a1e0c6ffaf455b965e1b6c80a"}
Nov 03 21:21:31.688491 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:31.688452    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-z5jk4" event={"ID":"a3134aab-ba99-4e98-974a-dc26c5966860","Type":"ContainerStarted","Data":"28ae09c83c013f6bfb39233a0cde7261ee3b4aa2c71eac6310565f90c5859e82"}
Nov 03 21:21:31.689032 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:31.688516    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-z5jk4"
Nov 03 21:21:31.690142 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:31.690116    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-vmp59" event={"ID":"36ec7139-4665-4a3c-ab8e-5ac33799ca58","Type":"ContainerStarted","Data":"cc76407a778d257f66e4df1fe51b3368d516c7133af6b56f378cd9ca4a4d08a7"}
Nov 03 21:21:31.690266 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:31.690253    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-vmp59"
Nov 03 21:21:31.804404 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:31.804336    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-z5jk4" podStartSLOduration=2.85736385 podStartE2EDuration="5.804320425s" totalImagesPullingTime="2.946956575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:21:31.804090466 +0000 UTC m=+406.298012342" watchObservedRunningTime="2026-11-03 21:21:31.804320425 +0000 UTC m=+406.298242296"
Nov 03 21:21:31.920529 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:31.920477    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-vmp59" podStartSLOduration=2.468109472 podStartE2EDuration="5.920462806s" totalImagesPullingTime="3.452353334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:21:31.918536873 +0000 UTC m=+406.412458748" watchObservedRunningTime="2026-11-03 21:21:31.920462806 +0000 UTC m=+406.414384709"
Nov 03 21:21:32.695926 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:32.695890    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-h2zs5" event={"ID":"53dadcdb-7f9d-4dd1-acb8-9321d2bf3ef3","Type":"ContainerStarted","Data":"7d25469c0fce638f88f5ae326f721f2cc5d6ba7f4df05c113d58b0f025689a61"}
Nov 03 21:21:32.696457 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:32.696178    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-h2zs5"
Nov 03 21:21:32.836285 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:32.836211    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-h2zs5" podStartSLOduration=2.765088573 podStartE2EDuration="6.836197527s" totalImagesPullingTime="4.071108954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:21:32.8332847 +0000 UTC m=+407.327206574" watchObservedRunningTime="2026-11-03 21:21:32.836197527 +0000 UTC m=+407.330119401"
Nov 03 21:21:40.949137 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:21:40.949096    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:21:42.713268 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:42.705416    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-z5jk4"
Nov 03 21:21:43.728591 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:43.728556    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-h2zs5"
Nov 03 21:21:47.922515 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:47.922470    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-blk95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:21:47.922911 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:47.922552    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95" podUID="48c6d17e-4e96-4837-8ec2-c1da7ecf30be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:21:49.779783 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:49.779748    5053 generic.go:356] "Generic (PLEG): container finished" podID="48c6d17e-4e96-4837-8ec2-c1da7ecf30be" containerID="7d7fb3b24a8708accbea047771ca706debce91a1360335ccf32223cc806171c7" exitCode=0
Nov 03 21:21:49.780297 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:49.779790    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95" event={"ID":"48c6d17e-4e96-4837-8ec2-c1da7ecf30be","Type":"ContainerDied","Data":"7d7fb3b24a8708accbea047771ca706debce91a1360335ccf32223cc806171c7"}
Nov 03 21:21:49.780297 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:49.780222    5053 scope.go:122] "RemoveContainer" containerID="7d7fb3b24a8708accbea047771ca706debce91a1360335ccf32223cc806171c7"
Nov 03 21:21:49.781434 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:49.781404    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:21:49.781517 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:49.781472    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:21:49.781517 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:49.781501    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:21:50.786492 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:50.786459    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95" event={"ID":"48c6d17e-4e96-4837-8ec2-c1da7ecf30be","Type":"ContainerStarted","Data":"396905b992d761068cfea1a2f506667ffb86bb769580394cb61b8a2b052b73b2"}
Nov 03 21:21:51.238850 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:51.238815    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:21:51.239126 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:51.239101    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-blk95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 03 21:21:51.239183 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:51.239165    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95" podUID="48c6d17e-4e96-4837-8ec2-c1da7ecf30be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 03 21:21:52.238500 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:52.238461    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-blk95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 03 21:21:52.238892 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:52.238531    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95" podUID="48c6d17e-4e96-4837-8ec2-c1da7ecf30be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 03 21:21:53.239552 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:53.239517    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-blk95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 03 21:21:53.239976 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:53.239574    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95" podUID="48c6d17e-4e96-4837-8ec2-c1da7ecf30be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 03 21:21:54.238636 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:54.238598    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-blk95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 03 21:21:54.238824 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:54.238656    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95" podUID="48c6d17e-4e96-4837-8ec2-c1da7ecf30be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 03 21:21:55.239040 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:55.239003    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-blk95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 03 21:21:55.239508 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:55.239098    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95" podUID="48c6d17e-4e96-4837-8ec2-c1da7ecf30be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 03 21:21:56.242051 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:56.242020    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:21:56.242541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:56.242342    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:21:56.243107 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:21:56.243093    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:22:02.701368 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:22:02.701336    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-vmp59"
Nov 03 21:22:10.926886 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:22:10.926848    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:22:40.926949 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:22:40.926912    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:23:10.927540 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:23:10.927501    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:23:18.199172 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:18.199134    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-6mcpf"]
Nov 03 21:23:18.203826 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:18.203794    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-6mcpf"
Nov 03 21:23:18.210979 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:18.210959    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-6mcpf\" (UID: \"8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-6mcpf"
Nov 03 21:23:18.211064 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:18.211018    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h62tw\" (UniqueName: \"kubernetes.io/projected/8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f-kube-api-access-h62tw\") pod \"ai-gateway-controller-5d8dddb676-6mcpf\" (UID: \"8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-6mcpf"
Nov 03 21:23:18.255141 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:18.255108    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-6mcpf"]
Nov 03 21:23:18.305850 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:18.305819    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 03 21:23:18.306081 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:18.305923    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 03 21:23:18.306203 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:18.306070    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-pxbws\""
Nov 03 21:23:18.311519 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:18.311490    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h62tw\" (UniqueName: \"kubernetes.io/projected/8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f-kube-api-access-h62tw\") pod \"ai-gateway-controller-5d8dddb676-6mcpf\" (UID: \"8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-6mcpf"
Nov 03 21:23:18.311657 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:18.311578    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-6mcpf\" (UID: \"8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-6mcpf"
Nov 03 21:23:18.314088 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:18.314065    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-6mcpf\" (UID: \"8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-6mcpf"
Nov 03 21:23:18.402026 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:18.401989    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h62tw\" (UniqueName: \"kubernetes.io/projected/8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f-kube-api-access-h62tw\") pod \"ai-gateway-controller-5d8dddb676-6mcpf\" (UID: \"8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-6mcpf"
Nov 03 21:23:18.515552 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:18.515457    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-6mcpf"
Nov 03 21:23:18.883772 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:18.883747    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-6mcpf"]
Nov 03 21:23:18.885920 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:23:18.885887    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e74e2f5_2bb2_45aa_afb6_5235ed99ac8f.slice/crio-291f09d2421978c9bf3023d7380ef413a09285c2d4184e7fa9a3e26546471cb4 WatchSource:0}: Error finding container 291f09d2421978c9bf3023d7380ef413a09285c2d4184e7fa9a3e26546471cb4: Status 404 returned error can't find the container with id 291f09d2421978c9bf3023d7380ef413a09285c2d4184e7fa9a3e26546471cb4
Nov 03 21:23:19.251858 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:19.251825    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-6mcpf" event={"ID":"8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f","Type":"ContainerStarted","Data":"291f09d2421978c9bf3023d7380ef413a09285c2d4184e7fa9a3e26546471cb4"}
Nov 03 21:23:22.273026 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:22.272991    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-6mcpf" event={"ID":"8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f","Type":"ContainerStarted","Data":"2a6820330368d81fdc52ad4fd83cbc6b446c3eab8dbc69c7f7713154f1642699"}
Nov 03 21:23:22.273442 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:22.273121    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-6mcpf"
Nov 03 21:23:22.386149 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:22.386094    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-6mcpf" podStartSLOduration=1.5480615389999999 podStartE2EDuration="4.386078638s" totalImagesPullingTime="2.838017099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:23:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:23:22.384472197 +0000 UTC m=+516.878394075" watchObservedRunningTime="2026-11-03 21:23:22.386078638 +0000 UTC m=+516.880000513"
Nov 03 21:23:33.281465 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:33.281433    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-6mcpf"
Nov 03 21:23:33.626008 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:33.625923    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-96cd8748-ck6m2"]
Nov 03 21:23:33.628716 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:33.628696    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-96cd8748-ck6m2"
Nov 03 21:23:33.686130 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:33.686095    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-96cd8748-ck6m2"]
Nov 03 21:23:33.744244 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:33.744193    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1461a40-86ef-4281-bb1f-b157e7b4e2f3-metrics-certs\") pod \"ai-gateway-controller-96cd8748-ck6m2\" (UID: \"a1461a40-86ef-4281-bb1f-b157e7b4e2f3\") " pod="opendatahub/ai-gateway-controller-96cd8748-ck6m2"
Nov 03 21:23:33.744444 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:33.744380    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kh6c\" (UniqueName: \"kubernetes.io/projected/a1461a40-86ef-4281-bb1f-b157e7b4e2f3-kube-api-access-8kh6c\") pod \"ai-gateway-controller-96cd8748-ck6m2\" (UID: \"a1461a40-86ef-4281-bb1f-b157e7b4e2f3\") " pod="opendatahub/ai-gateway-controller-96cd8748-ck6m2"
Nov 03 21:23:33.845570 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:33.845529    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1461a40-86ef-4281-bb1f-b157e7b4e2f3-metrics-certs\") pod \"ai-gateway-controller-96cd8748-ck6m2\" (UID: \"a1461a40-86ef-4281-bb1f-b157e7b4e2f3\") " pod="opendatahub/ai-gateway-controller-96cd8748-ck6m2"
Nov 03 21:23:33.845746 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:33.845614    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kh6c\" (UniqueName: \"kubernetes.io/projected/a1461a40-86ef-4281-bb1f-b157e7b4e2f3-kube-api-access-8kh6c\") pod \"ai-gateway-controller-96cd8748-ck6m2\" (UID: \"a1461a40-86ef-4281-bb1f-b157e7b4e2f3\") " pod="opendatahub/ai-gateway-controller-96cd8748-ck6m2"
Nov 03 21:23:33.848099 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:33.848072    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1461a40-86ef-4281-bb1f-b157e7b4e2f3-metrics-certs\") pod \"ai-gateway-controller-96cd8748-ck6m2\" (UID: \"a1461a40-86ef-4281-bb1f-b157e7b4e2f3\") " pod="opendatahub/ai-gateway-controller-96cd8748-ck6m2"
Nov 03 21:23:33.905435 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:33.905361    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kh6c\" (UniqueName: \"kubernetes.io/projected/a1461a40-86ef-4281-bb1f-b157e7b4e2f3-kube-api-access-8kh6c\") pod \"ai-gateway-controller-96cd8748-ck6m2\" (UID: \"a1461a40-86ef-4281-bb1f-b157e7b4e2f3\") " pod="opendatahub/ai-gateway-controller-96cd8748-ck6m2"
Nov 03 21:23:33.940623 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:33.940594    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-96cd8748-ck6m2"
Nov 03 21:23:34.307739 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:34.307689    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-96cd8748-ck6m2"]
Nov 03 21:23:34.309893 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:23:34.309853    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1461a40_86ef_4281_bb1f_b157e7b4e2f3.slice/crio-f7e9b8a16b24db6c17275f0bb7f5eb3d757a048cfa768e836f059b8d06c51abe WatchSource:0}: Error finding container f7e9b8a16b24db6c17275f0bb7f5eb3d757a048cfa768e836f059b8d06c51abe: Status 404 returned error can't find the container with id f7e9b8a16b24db6c17275f0bb7f5eb3d757a048cfa768e836f059b8d06c51abe
Nov 03 21:23:34.331916 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:34.331886    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-96cd8748-ck6m2" event={"ID":"a1461a40-86ef-4281-bb1f-b157e7b4e2f3","Type":"ContainerStarted","Data":"f7e9b8a16b24db6c17275f0bb7f5eb3d757a048cfa768e836f059b8d06c51abe"}
Nov 03 21:23:35.339139 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:35.339103    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-96cd8748-ck6m2" event={"ID":"a1461a40-86ef-4281-bb1f-b157e7b4e2f3","Type":"ContainerStarted","Data":"78e2aea79e83fe2d51bbdc6563ed487a662f1e355adca8827e22b353565fe6d9"}
Nov 03 21:23:35.339572 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:35.339220    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-96cd8748-ck6m2"
Nov 03 21:23:35.453038 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:35.452989    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-96cd8748-ck6m2" podStartSLOduration=1.6551859119999999 podStartE2EDuration="2.452974612s" totalImagesPullingTime="797.7887ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:23:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:23:35.451629267 +0000 UTC m=+529.945551133" watchObservedRunningTime="2026-11-03 21:23:35.452974612 +0000 UTC m=+529.946896487"
Nov 03 21:23:40.926989 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:23:40.926951    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:23:46.346744 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:46.346709    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-96cd8748-ck6m2"
Nov 03 21:23:46.511051 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:46.511018    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-6mcpf"]
Nov 03 21:23:46.511307 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:46.511282    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-6mcpf" podUID="8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f" containerName="manager" containerID="cri-o://2a6820330368d81fdc52ad4fd83cbc6b446c3eab8dbc69c7f7713154f1642699" gracePeriod=30
Nov 03 21:23:46.759528 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:46.759506    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-6mcpf"
Nov 03 21:23:46.856441 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:46.856405    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f-metrics-certs\") pod \"8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f\" (UID: \"8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f\") "
Nov 03 21:23:46.856638 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:46.856469    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f-kube-api-access-h62tw\" (UniqueName: \"kubernetes.io/projected/8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f-kube-api-access-h62tw\") pod \"8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f\" (UID: \"8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f\") "
Nov 03 21:23:46.858864 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:46.858832    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f-kube-api-access-h62tw" pod "8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f" (UID: "8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f"). InnerVolumeSpecName "kube-api-access-h62tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:23:46.858968 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:46.858888    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f-metrics-certs" pod "8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f" (UID: "8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:23:46.957817 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:46.957789    5053 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f-metrics-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:23:46.957974 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:46.957818    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h62tw\" (UniqueName: \"kubernetes.io/projected/8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f-kube-api-access-h62tw\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:23:47.394193 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:47.394109    5053 generic.go:356] "Generic (PLEG): container finished" podID="8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f" containerID="2a6820330368d81fdc52ad4fd83cbc6b446c3eab8dbc69c7f7713154f1642699" exitCode=0
Nov 03 21:23:47.394193 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:47.394149    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-6mcpf"
Nov 03 21:23:47.394193 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:47.394181    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-6mcpf" event={"ID":"8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f","Type":"ContainerDied","Data":"2a6820330368d81fdc52ad4fd83cbc6b446c3eab8dbc69c7f7713154f1642699"}
Nov 03 21:23:47.394685 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:47.394249    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-6mcpf" event={"ID":"8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f","Type":"ContainerDied","Data":"291f09d2421978c9bf3023d7380ef413a09285c2d4184e7fa9a3e26546471cb4"}
Nov 03 21:23:47.394685 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:47.394275    5053 scope.go:122] "RemoveContainer" containerID="2a6820330368d81fdc52ad4fd83cbc6b446c3eab8dbc69c7f7713154f1642699"
Nov 03 21:23:47.403251 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:47.403217    5053 scope.go:122] "RemoveContainer" containerID="2a6820330368d81fdc52ad4fd83cbc6b446c3eab8dbc69c7f7713154f1642699"
Nov 03 21:23:47.403579 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:23:47.403555    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a6820330368d81fdc52ad4fd83cbc6b446c3eab8dbc69c7f7713154f1642699\": container with ID starting with 2a6820330368d81fdc52ad4fd83cbc6b446c3eab8dbc69c7f7713154f1642699 not found: ID does not exist" containerID="2a6820330368d81fdc52ad4fd83cbc6b446c3eab8dbc69c7f7713154f1642699"
Nov 03 21:23:47.403634 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:47.403581    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a6820330368d81fdc52ad4fd83cbc6b446c3eab8dbc69c7f7713154f1642699"} err="failed to get container status \"2a6820330368d81fdc52ad4fd83cbc6b446c3eab8dbc69c7f7713154f1642699\": rpc error: code = NotFound desc = could not find container \"2a6820330368d81fdc52ad4fd83cbc6b446c3eab8dbc69c7f7713154f1642699\": container with ID starting with 2a6820330368d81fdc52ad4fd83cbc6b446c3eab8dbc69c7f7713154f1642699 not found: ID does not exist"
Nov 03 21:23:47.556737 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:47.556696    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-6mcpf"]
Nov 03 21:23:47.606467 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:47.606437    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-6mcpf"]
Nov 03 21:23:47.666150 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:23:47.666074    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f" path="/var/lib/kubelet/pods/8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f/volumes"
Nov 03 21:24:03.556726 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.556687    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"]
Nov 03 21:24:03.557150 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.557137    5053 cpu_manager.go:414] "containerMap: removing container" podUID="8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f" containerName="manager"
Nov 03 21:24:03.557197 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.557151    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f" containerName="manager"
Nov 03 21:24:03.557230 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.557211    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e74e2f5-2bb2-45aa-afb6-5235ed99ac8f" containerName="manager"
Nov 03 21:24:03.560405 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.560389    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"
Nov 03 21:24:03.665076 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.665041    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 21:24:03.665301 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.665121    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 21:24:03.665301 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.665164    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-x8lxr\""
Nov 03 21:24:03.665301 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.665168    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 03 21:24:03.667100 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.667074    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"]
Nov 03 21:24:03.708583 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.708549    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a2bbba0-9cc7-4f82-9803-abfc59037289-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x8csc\" (UID: \"5a2bbba0-9cc7-4f82-9803-abfc59037289\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"
Nov 03 21:24:03.708766 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.708597    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qfsz\" (UniqueName: \"kubernetes.io/projected/5a2bbba0-9cc7-4f82-9803-abfc59037289-kube-api-access-6qfsz\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x8csc\" (UID: \"5a2bbba0-9cc7-4f82-9803-abfc59037289\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"
Nov 03 21:24:03.708766 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.708627    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a2bbba0-9cc7-4f82-9803-abfc59037289-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x8csc\" (UID: \"5a2bbba0-9cc7-4f82-9803-abfc59037289\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"
Nov 03 21:24:03.708766 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.708696    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a2bbba0-9cc7-4f82-9803-abfc59037289-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x8csc\" (UID: \"5a2bbba0-9cc7-4f82-9803-abfc59037289\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"
Nov 03 21:24:03.708766 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.708722    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a2bbba0-9cc7-4f82-9803-abfc59037289-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x8csc\" (UID: \"5a2bbba0-9cc7-4f82-9803-abfc59037289\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"
Nov 03 21:24:03.708911 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.708838    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a2bbba0-9cc7-4f82-9803-abfc59037289-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x8csc\" (UID: \"5a2bbba0-9cc7-4f82-9803-abfc59037289\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"
Nov 03 21:24:03.810064 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.809975    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a2bbba0-9cc7-4f82-9803-abfc59037289-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x8csc\" (UID: \"5a2bbba0-9cc7-4f82-9803-abfc59037289\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"
Nov 03 21:24:03.810064 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.810041    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qfsz\" (UniqueName: \"kubernetes.io/projected/5a2bbba0-9cc7-4f82-9803-abfc59037289-kube-api-access-6qfsz\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x8csc\" (UID: \"5a2bbba0-9cc7-4f82-9803-abfc59037289\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"
Nov 03 21:24:03.810349 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.810068    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a2bbba0-9cc7-4f82-9803-abfc59037289-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x8csc\" (UID: \"5a2bbba0-9cc7-4f82-9803-abfc59037289\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"
Nov 03 21:24:03.810349 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.810103    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a2bbba0-9cc7-4f82-9803-abfc59037289-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x8csc\" (UID: \"5a2bbba0-9cc7-4f82-9803-abfc59037289\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"
Nov 03 21:24:03.810349 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.810137    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a2bbba0-9cc7-4f82-9803-abfc59037289-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x8csc\" (UID: \"5a2bbba0-9cc7-4f82-9803-abfc59037289\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"
Nov 03 21:24:03.810349 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.810217    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a2bbba0-9cc7-4f82-9803-abfc59037289-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x8csc\" (UID: \"5a2bbba0-9cc7-4f82-9803-abfc59037289\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"
Nov 03 21:24:03.810870 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.810845    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a2bbba0-9cc7-4f82-9803-abfc59037289-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x8csc\" (UID: \"5a2bbba0-9cc7-4f82-9803-abfc59037289\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"
Nov 03 21:24:03.810978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.810878    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a2bbba0-9cc7-4f82-9803-abfc59037289-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x8csc\" (UID: \"5a2bbba0-9cc7-4f82-9803-abfc59037289\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"
Nov 03 21:24:03.811018 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.810962    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a2bbba0-9cc7-4f82-9803-abfc59037289-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x8csc\" (UID: \"5a2bbba0-9cc7-4f82-9803-abfc59037289\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"
Nov 03 21:24:03.812585 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.812559    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a2bbba0-9cc7-4f82-9803-abfc59037289-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x8csc\" (UID: \"5a2bbba0-9cc7-4f82-9803-abfc59037289\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"
Nov 03 21:24:03.812846 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.812829    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a2bbba0-9cc7-4f82-9803-abfc59037289-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x8csc\" (UID: \"5a2bbba0-9cc7-4f82-9803-abfc59037289\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"
Nov 03 21:24:03.866045 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.866008    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qfsz\" (UniqueName: \"kubernetes.io/projected/5a2bbba0-9cc7-4f82-9803-abfc59037289-kube-api-access-6qfsz\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x8csc\" (UID: \"5a2bbba0-9cc7-4f82-9803-abfc59037289\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"
Nov 03 21:24:03.872717 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:03.872690    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"
Nov 03 21:24:04.250837 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:04.250813    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"]
Nov 03 21:24:04.252435 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:24:04.252407    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a2bbba0_9cc7_4f82_9803_abfc59037289.slice/crio-36e3f1dde1c69fe7f4fee2d62fb9b62cf11e352c26e05681de841cb15551423d WatchSource:0}: Error finding container 36e3f1dde1c69fe7f4fee2d62fb9b62cf11e352c26e05681de841cb15551423d: Status 404 returned error can't find the container with id 36e3f1dde1c69fe7f4fee2d62fb9b62cf11e352c26e05681de841cb15551423d
Nov 03 21:24:04.481892 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:04.481859    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc" event={"ID":"5a2bbba0-9cc7-4f82-9803-abfc59037289","Type":"ContainerStarted","Data":"36e3f1dde1c69fe7f4fee2d62fb9b62cf11e352c26e05681de841cb15551423d"}
Nov 03 21:24:06.494644 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:06.494605    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc" event={"ID":"5a2bbba0-9cc7-4f82-9803-abfc59037289","Type":"ContainerStarted","Data":"bb0410eaee14e0b4c745d1c9c62e8ff30e1b468b0d94a51bcfebc6bfd24a2ccc"}
Nov 03 21:24:06.616152 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:06.616073    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc" podStartSLOduration=2.098583459 podStartE2EDuration="3.616054566s" totalImagesPullingTime="1.517471107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:06.612991484 +0000 UTC m=+561.106913363" watchObservedRunningTime="2026-11-03 21:24:06.616054566 +0000 UTC m=+561.109976443"
Nov 03 21:24:06.873251 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:06.873149    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"
Nov 03 21:24:10.927413 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:24:10.927375    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:24:12.126447 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:12.126409    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"]
Nov 03 21:24:12.130882 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:12.130848    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"
Nov 03 21:24:12.232586 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:12.232559    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 03 21:24:12.235700 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:12.235670    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"]
Nov 03 21:24:12.291177 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:12.291141    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aad4a0a-f17b-48bf-91bd-7f3e193e83a8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-btvm4\" (UID: \"1aad4a0a-f17b-48bf-91bd-7f3e193e83a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"
Nov 03 21:24:12.291373 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:12.291191    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm44m\" (UniqueName: \"kubernetes.io/projected/1aad4a0a-f17b-48bf-91bd-7f3e193e83a8-kube-api-access-nm44m\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-btvm4\" (UID: \"1aad4a0a-f17b-48bf-91bd-7f3e193e83a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"
Nov 03 21:24:12.291373 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:12.291349    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aad4a0a-f17b-48bf-91bd-7f3e193e83a8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-btvm4\" (UID: \"1aad4a0a-f17b-48bf-91bd-7f3e193e83a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"
Nov 03 21:24:12.291454 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:12.291380    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aad4a0a-f17b-48bf-91bd-7f3e193e83a8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-btvm4\" (UID: \"1aad4a0a-f17b-48bf-91bd-7f3e193e83a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"
Nov 03 21:24:12.291454 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:12.291419    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aad4a0a-f17b-48bf-91bd-7f3e193e83a8-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-btvm4\" (UID: \"1aad4a0a-f17b-48bf-91bd-7f3e193e83a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"
Nov 03 21:24:12.291514 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:12.291493    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aad4a0a-f17b-48bf-91bd-7f3e193e83a8-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-btvm4\" (UID: \"1aad4a0a-f17b-48bf-91bd-7f3e193e83a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"
Nov 03 21:24:12.392431 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:12.392334    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aad4a0a-f17b-48bf-91bd-7f3e193e83a8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-btvm4\" (UID: \"1aad4a0a-f17b-48bf-91bd-7f3e193e83a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"
Nov 03 21:24:12.392431 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:12.392383    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm44m\" (UniqueName: \"kubernetes.io/projected/1aad4a0a-f17b-48bf-91bd-7f3e193e83a8-kube-api-access-nm44m\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-btvm4\" (UID: \"1aad4a0a-f17b-48bf-91bd-7f3e193e83a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"
Nov 03 21:24:12.392684 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:12.392504    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aad4a0a-f17b-48bf-91bd-7f3e193e83a8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-btvm4\" (UID: \"1aad4a0a-f17b-48bf-91bd-7f3e193e83a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"
Nov 03 21:24:12.392684 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:12.392543    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aad4a0a-f17b-48bf-91bd-7f3e193e83a8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-btvm4\" (UID: \"1aad4a0a-f17b-48bf-91bd-7f3e193e83a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"
Nov 03 21:24:12.392684 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:12.392560    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aad4a0a-f17b-48bf-91bd-7f3e193e83a8-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-btvm4\" (UID: \"1aad4a0a-f17b-48bf-91bd-7f3e193e83a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"
Nov 03 21:24:12.392684 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:12.392614    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aad4a0a-f17b-48bf-91bd-7f3e193e83a8-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-btvm4\" (UID: \"1aad4a0a-f17b-48bf-91bd-7f3e193e83a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"
Nov 03 21:24:12.392885 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:12.392869    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aad4a0a-f17b-48bf-91bd-7f3e193e83a8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-btvm4\" (UID: \"1aad4a0a-f17b-48bf-91bd-7f3e193e83a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"
Nov 03 21:24:12.392974 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:12.392948    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aad4a0a-f17b-48bf-91bd-7f3e193e83a8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-btvm4\" (UID: \"1aad4a0a-f17b-48bf-91bd-7f3e193e83a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"
Nov 03 21:24:12.393082 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:12.393030    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aad4a0a-f17b-48bf-91bd-7f3e193e83a8-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-btvm4\" (UID: \"1aad4a0a-f17b-48bf-91bd-7f3e193e83a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"
Nov 03 21:24:12.394923 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:12.394906    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aad4a0a-f17b-48bf-91bd-7f3e193e83a8-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-btvm4\" (UID: \"1aad4a0a-f17b-48bf-91bd-7f3e193e83a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"
Nov 03 21:24:12.395177 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:12.395159    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aad4a0a-f17b-48bf-91bd-7f3e193e83a8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-btvm4\" (UID: \"1aad4a0a-f17b-48bf-91bd-7f3e193e83a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"
Nov 03 21:24:12.448845 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:12.448812    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm44m\" (UniqueName: \"kubernetes.io/projected/1aad4a0a-f17b-48bf-91bd-7f3e193e83a8-kube-api-access-nm44m\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-btvm4\" (UID: \"1aad4a0a-f17b-48bf-91bd-7f3e193e83a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"
Nov 03 21:24:12.745764 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:12.745725    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"
Nov 03 21:24:13.117190 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:13.117167    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"]
Nov 03 21:24:13.118867 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:24:13.118832    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aad4a0a_f17b_48bf_91bd_7f3e193e83a8.slice/crio-2e41f033445370b5e44fa22c1c265bcd585c0b2dbc18599af499daed3328718a WatchSource:0}: Error finding container 2e41f033445370b5e44fa22c1c265bcd585c0b2dbc18599af499daed3328718a: Status 404 returned error can't find the container with id 2e41f033445370b5e44fa22c1c265bcd585c0b2dbc18599af499daed3328718a
Nov 03 21:24:13.538116 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:13.538079    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4" event={"ID":"1aad4a0a-f17b-48bf-91bd-7f3e193e83a8","Type":"ContainerStarted","Data":"2e41f033445370b5e44fa22c1c265bcd585c0b2dbc18599af499daed3328718a"}
Nov 03 21:24:13.873432 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:13.873393    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"
Nov 03 21:24:13.886866 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:13.886831    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"
Nov 03 21:24:14.546287 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:14.545281    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4" event={"ID":"1aad4a0a-f17b-48bf-91bd-7f3e193e83a8","Type":"ContainerStarted","Data":"5538d5cace6dc8dbfd5e02dc1ca03a0d620ea1c9b54dd83cf6487e8e6321a4b7"}
Nov 03 21:24:14.556447 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:14.556421    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x8csc"
Nov 03 21:24:14.659376 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:14.659308    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4" podStartSLOduration=2.02575521 podStartE2EDuration="2.659287637s" totalImagesPullingTime="633.532427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:14.659035702 +0000 UTC m=+569.152957576" watchObservedRunningTime="2026-11-03 21:24:14.659287637 +0000 UTC m=+569.153209511"
Nov 03 21:24:14.746651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:14.746608    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"
Nov 03 21:24:15.303905 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:15.303851    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"]
Nov 03 21:24:15.307834 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:15.307810    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"
Nov 03 21:24:15.365528 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:15.365493    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"]
Nov 03 21:24:15.411487 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:15.411449    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 03 21:24:15.422922 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:15.422886    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h\" (UID: \"1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"
Nov 03 21:24:15.423062 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:15.422929    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h\" (UID: \"1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"
Nov 03 21:24:15.423062 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:15.423009    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h\" (UID: \"1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"
Nov 03 21:24:15.423147 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:15.423106    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h\" (UID: \"1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"
Nov 03 21:24:15.423147 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:15.423139    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9n5b\" (UniqueName: \"kubernetes.io/projected/1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2-kube-api-access-c9n5b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h\" (UID: \"1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"
Nov 03 21:24:15.423226 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:15.423161    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h\" (UID: \"1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"
Nov 03 21:24:15.523775 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:15.523739    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h\" (UID: \"1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"
Nov 03 21:24:15.523775 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:15.523777    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h\" (UID: \"1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"
Nov 03 21:24:15.524204 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:15.523799    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h\" (UID: \"1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"
Nov 03 21:24:15.524204 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:15.523835    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h\" (UID: \"1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"
Nov 03 21:24:15.524204 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:15.523855    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9n5b\" (UniqueName: \"kubernetes.io/projected/1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2-kube-api-access-c9n5b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h\" (UID: \"1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"
Nov 03 21:24:15.524204 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:15.523872    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h\" (UID: \"1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"
Nov 03 21:24:15.524451 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:15.524261    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h\" (UID: \"1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"
Nov 03 21:24:15.524451 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:15.524342    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h\" (UID: \"1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"
Nov 03 21:24:15.524451 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:15.524352    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h\" (UID: \"1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"
Nov 03 21:24:15.526294 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:15.526276    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h\" (UID: \"1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"
Nov 03 21:24:15.526454 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:15.526438    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h\" (UID: \"1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"
Nov 03 21:24:15.580000 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:15.579918    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9n5b\" (UniqueName: \"kubernetes.io/projected/1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2-kube-api-access-c9n5b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h\" (UID: \"1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"
Nov 03 21:24:15.622069 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:15.622035    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"
Nov 03 21:24:16.004032 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:16.004002    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"]
Nov 03 21:24:16.004182 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:24:16.004161    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4c0b5f_ee49_4cb5_bc1e_fe80052bb1e2.slice/crio-cf02e01c979048f856cca6606c4de0b52693ddd0c8fe5308195e4c0b673132c5 WatchSource:0}: Error finding container cf02e01c979048f856cca6606c4de0b52693ddd0c8fe5308195e4c0b673132c5: Status 404 returned error can't find the container with id cf02e01c979048f856cca6606c4de0b52693ddd0c8fe5308195e4c0b673132c5
Nov 03 21:24:16.558031 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:16.557994    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h" event={"ID":"1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2","Type":"ContainerStarted","Data":"cf02e01c979048f856cca6606c4de0b52693ddd0c8fe5308195e4c0b673132c5"}
Nov 03 21:24:17.564891 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:17.564856    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h" event={"ID":"1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2","Type":"ContainerStarted","Data":"ffc29276e2b6cca590487b50e23d2d917ddaa8b66c39c9acb8f90ee1cfec1b2f"}
Nov 03 21:24:17.622651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:17.622622    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"
Nov 03 21:24:17.678250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:17.678180    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h" podStartSLOduration=2.040145964 podStartE2EDuration="2.678166469s" totalImagesPullingTime="638.020505ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:17.677416448 +0000 UTC m=+572.171338325" watchObservedRunningTime="2026-11-03 21:24:17.678166469 +0000 UTC m=+572.172088378"
Nov 03 21:24:21.949460 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:21.949432    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"]
Nov 03 21:24:21.953285 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:21.953264    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"
Nov 03 21:24:21.973387 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:21.973366    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24f5f506-2f6d-40f9-a692-5e62af63d3d1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75\" (UID: \"24f5f506-2f6d-40f9-a692-5e62af63d3d1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"
Nov 03 21:24:21.973496 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:21.973397    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24f5f506-2f6d-40f9-a692-5e62af63d3d1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75\" (UID: \"24f5f506-2f6d-40f9-a692-5e62af63d3d1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"
Nov 03 21:24:21.973496 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:21.973432    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg825\" (UniqueName: \"kubernetes.io/projected/24f5f506-2f6d-40f9-a692-5e62af63d3d1-kube-api-access-zg825\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75\" (UID: \"24f5f506-2f6d-40f9-a692-5e62af63d3d1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"
Nov 03 21:24:21.973579 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:21.973528    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24f5f506-2f6d-40f9-a692-5e62af63d3d1-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75\" (UID: \"24f5f506-2f6d-40f9-a692-5e62af63d3d1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"
Nov 03 21:24:21.973635 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:21.973597    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24f5f506-2f6d-40f9-a692-5e62af63d3d1-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75\" (UID: \"24f5f506-2f6d-40f9-a692-5e62af63d3d1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"
Nov 03 21:24:21.973711 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:21.973692    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24f5f506-2f6d-40f9-a692-5e62af63d3d1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75\" (UID: \"24f5f506-2f6d-40f9-a692-5e62af63d3d1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"
Nov 03 21:24:22.009513 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:22.009483    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"]
Nov 03 21:24:22.056437 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:22.056402    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 03 21:24:22.074429 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:22.074402    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24f5f506-2f6d-40f9-a692-5e62af63d3d1-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75\" (UID: \"24f5f506-2f6d-40f9-a692-5e62af63d3d1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"
Nov 03 21:24:22.074568 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:22.074489    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24f5f506-2f6d-40f9-a692-5e62af63d3d1-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75\" (UID: \"24f5f506-2f6d-40f9-a692-5e62af63d3d1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"
Nov 03 21:24:22.074568 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:22.074519    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24f5f506-2f6d-40f9-a692-5e62af63d3d1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75\" (UID: \"24f5f506-2f6d-40f9-a692-5e62af63d3d1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"
Nov 03 21:24:22.074568 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:22.074559    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24f5f506-2f6d-40f9-a692-5e62af63d3d1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75\" (UID: \"24f5f506-2f6d-40f9-a692-5e62af63d3d1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"
Nov 03 21:24:22.074707 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:22.074576    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24f5f506-2f6d-40f9-a692-5e62af63d3d1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75\" (UID: \"24f5f506-2f6d-40f9-a692-5e62af63d3d1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"
Nov 03 21:24:22.074749 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:22.074704    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg825\" (UniqueName: \"kubernetes.io/projected/24f5f506-2f6d-40f9-a692-5e62af63d3d1-kube-api-access-zg825\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75\" (UID: \"24f5f506-2f6d-40f9-a692-5e62af63d3d1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"
Nov 03 21:24:22.074924 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:22.074901    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24f5f506-2f6d-40f9-a692-5e62af63d3d1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75\" (UID: \"24f5f506-2f6d-40f9-a692-5e62af63d3d1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"
Nov 03 21:24:22.075066 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:22.074955    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24f5f506-2f6d-40f9-a692-5e62af63d3d1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75\" (UID: \"24f5f506-2f6d-40f9-a692-5e62af63d3d1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"
Nov 03 21:24:22.075129 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:22.075002    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24f5f506-2f6d-40f9-a692-5e62af63d3d1-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75\" (UID: \"24f5f506-2f6d-40f9-a692-5e62af63d3d1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"
Nov 03 21:24:22.076897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:22.076876    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24f5f506-2f6d-40f9-a692-5e62af63d3d1-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75\" (UID: \"24f5f506-2f6d-40f9-a692-5e62af63d3d1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"
Nov 03 21:24:22.077279 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:22.077257    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24f5f506-2f6d-40f9-a692-5e62af63d3d1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75\" (UID: \"24f5f506-2f6d-40f9-a692-5e62af63d3d1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"
Nov 03 21:24:22.130912 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:22.130874    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg825\" (UniqueName: \"kubernetes.io/projected/24f5f506-2f6d-40f9-a692-5e62af63d3d1-kube-api-access-zg825\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75\" (UID: \"24f5f506-2f6d-40f9-a692-5e62af63d3d1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"
Nov 03 21:24:22.266385 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:22.266293    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"
Nov 03 21:24:22.639951 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:22.639924    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"]
Nov 03 21:24:22.641775 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:24:22.641736    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f5f506_2f6d_40f9_a692_5e62af63d3d1.slice/crio-3904429c921612670b56014f40f207c19460978cd36dce3cdb61977d580e5c99 WatchSource:0}: Error finding container 3904429c921612670b56014f40f207c19460978cd36dce3cdb61977d580e5c99: Status 404 returned error can't find the container with id 3904429c921612670b56014f40f207c19460978cd36dce3cdb61977d580e5c99
Nov 03 21:24:22.746326 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:22.746291    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"
Nov 03 21:24:22.758569 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:22.758544    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"
Nov 03 21:24:23.598448 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:23.598348    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75" event={"ID":"24f5f506-2f6d-40f9-a692-5e62af63d3d1","Type":"ContainerStarted","Data":"3cb42198c5a939bbbe4dcbbc0b36021d192e3a50b2faf426b2ba3c5e8aea05e5"}
Nov 03 21:24:23.598448 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:23.598431    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75" event={"ID":"24f5f506-2f6d-40f9-a692-5e62af63d3d1","Type":"ContainerStarted","Data":"3904429c921612670b56014f40f207c19460978cd36dce3cdb61977d580e5c99"}
Nov 03 21:24:23.609217 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:23.609193    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-btvm4"
Nov 03 21:24:23.714367 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:23.714308    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75" podStartSLOduration=2.096772027 podStartE2EDuration="2.714288824s" totalImagesPullingTime="617.516797ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:23.710977977 +0000 UTC m=+578.204899852" watchObservedRunningTime="2026-11-03 21:24:23.714288824 +0000 UTC m=+578.208210699"
Nov 03 21:24:24.266707 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:24.266677    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"
Nov 03 21:24:25.622810 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:25.622720    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"
Nov 03 21:24:25.635253 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:25.635212    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"
Nov 03 21:24:26.626078 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:26.626053    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h"
Nov 03 21:24:32.267422 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:32.267382    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"
Nov 03 21:24:32.279374 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:32.279347    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"
Nov 03 21:24:32.658906 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:32.658827    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75"
Nov 03 21:24:40.927822 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:24:40.927780    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:24:43.492870 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.492834    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"]
Nov 03 21:24:43.499064 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.499039    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.584380 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.584343    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.584380 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.584381    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.584582 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.584408    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.584582 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.584528    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79w2h\" (UniqueName: \"kubernetes.io/projected/909e67ce-5511-4f93-91eb-17f6a87acd84-kube-api-access-79w2h\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.584582 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.584572    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/909e67ce-5511-4f93-91eb-17f6a87acd84-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.584688 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.584591    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.584688 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.584665    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.584761 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.584689    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.584799 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.584777    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.597570 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.597536    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"]
Nov 03 21:24:43.685324 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.685290    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.685495 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.685332    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.685495 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.685366    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.685495 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.685401    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.685495 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.685416    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.685673 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.685514    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.685673 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.685588    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79w2h\" (UniqueName: \"kubernetes.io/projected/909e67ce-5511-4f93-91eb-17f6a87acd84-kube-api-access-79w2h\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.685673 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.685650    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/909e67ce-5511-4f93-91eb-17f6a87acd84-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.685814 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.685676    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.685814 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.685727    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.685814 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.685755    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.685961 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.685829    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.686625 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.686592    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.689270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.686921    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/909e67ce-5511-4f93-91eb-17f6a87acd84-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.692000 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.691970    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.694700 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.694676    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.741503 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.741468    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.741715 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.741692    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79w2h\" (UniqueName: \"kubernetes.io/projected/909e67ce-5511-4f93-91eb-17f6a87acd84-kube-api-access-79w2h\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8jgdx\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:43.812293 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:43.812193    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:44.283451 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:44.283423    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"]
Nov 03 21:24:44.285256 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:24:44.285211    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909e67ce_5511_4f93_91eb_17f6a87acd84.slice/crio-a8dd8c8f97e3c9da0625086afd7f2d882c0433aa11c5efdc9b3de794c36a2445 WatchSource:0}: Error finding container a8dd8c8f97e3c9da0625086afd7f2d882c0433aa11c5efdc9b3de794c36a2445: Status 404 returned error can't find the container with id a8dd8c8f97e3c9da0625086afd7f2d882c0433aa11c5efdc9b3de794c36a2445
Nov 03 21:24:44.287296 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:44.287262    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:24:44.287358 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:44.287348    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:24:44.287389 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:44.287377    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:24:44.715702 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:44.715662    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx" event={"ID":"909e67ce-5511-4f93-91eb-17f6a87acd84","Type":"ContainerStarted","Data":"6f2ec128abc20fdaad9e2fb701563bc0f3c7e727bf1048449095aaadec934b84"}
Nov 03 21:24:44.716066 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:44.715724    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx" event={"ID":"909e67ce-5511-4f93-91eb-17f6a87acd84","Type":"ContainerStarted","Data":"a8dd8c8f97e3c9da0625086afd7f2d882c0433aa11c5efdc9b3de794c36a2445"}
Nov 03 21:24:44.812905 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:44.812871    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:44.880479 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:44.880424    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx" podStartSLOduration=1.880407567 podStartE2EDuration="1.880407567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:44.830395367 +0000 UTC m=+599.324317244" watchObservedRunningTime="2026-11-03 21:24:44.880407567 +0000 UTC m=+599.374329442"
Nov 03 21:24:45.741673 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:45.741640    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 21:24:45.742016 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:45.741789    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 21:24:45.749069 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:45.749045    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:24:45.749378 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:45.749362    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:24:45.827105 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:45.827068    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8jgdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 03 21:24:45.827292 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:45.827124    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx" podUID="909e67ce-5511-4f93-91eb-17f6a87acd84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 03 21:24:46.821670 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:46.821630    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8jgdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 03 21:24:46.822322 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:46.822288    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx" podUID="909e67ce-5511-4f93-91eb-17f6a87acd84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 03 21:24:47.821266 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:47.819646    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8jgdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 03 21:24:47.821266 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:47.819711    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx" podUID="909e67ce-5511-4f93-91eb-17f6a87acd84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 03 21:24:48.817626 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:48.817588    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8jgdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 03 21:24:48.818944 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:48.818911    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx" podUID="909e67ce-5511-4f93-91eb-17f6a87acd84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 03 21:24:49.821267 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:49.820677    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8jgdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 03 21:24:49.821267 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:49.820756    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx" podUID="909e67ce-5511-4f93-91eb-17f6a87acd84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 03 21:24:50.812983 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:50.812942    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8jgdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 03 21:24:50.813194 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:50.813033    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx" podUID="909e67ce-5511-4f93-91eb-17f6a87acd84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 03 21:24:52.295321 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:52.295282    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:52.295935 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:52.295914    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:24:52.296922 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:24:52.296904    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:25:10.926947 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:25:10.926904    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:25:25.376315 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.376277    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"]
Nov 03 21:25:25.380221 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.380199    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.467450 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.467400    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-35257bc0-openshift-default-dockercfg-fjl5p\""
Nov 03 21:25:25.485256 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.485205    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"]
Nov 03 21:25:25.518305 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.517516    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-token\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.518305 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.517601    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-data\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.518305 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.517648    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-workload-socket\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.518305 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.517683    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-podinfo\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.518305 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.517748    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7pts\" (UniqueName: \"kubernetes.io/projected/610101d0-cf94-41b6-a21e-18e652eb9b03-kube-api-access-m7pts\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.518305 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.517796    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-credential-socket\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.518305 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.517820    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/610101d0-cf94-41b6-a21e-18e652eb9b03-istiod-ca-cert\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.518305 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.517863    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-workload-certs\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.518305 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.517919    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-envoy\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.619127 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.619095    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7pts\" (UniqueName: \"kubernetes.io/projected/610101d0-cf94-41b6-a21e-18e652eb9b03-kube-api-access-m7pts\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.619351 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.619151    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-credential-socket\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.619351 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.619174    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/610101d0-cf94-41b6-a21e-18e652eb9b03-istiod-ca-cert\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.619351 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.619213    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-workload-certs\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.619351 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.619274    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-envoy\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.619351 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.619299    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-token\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.619622 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.619353    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-data\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.619622 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.619392    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-workload-socket\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.619622 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.619427    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-podinfo\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.619768 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.619628    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-credential-socket\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.619768 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.619698    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-workload-certs\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.619932 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.619909    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-data\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.619999 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.619971    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-workload-socket\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.620117 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.620099    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/610101d0-cf94-41b6-a21e-18e652eb9b03-istiod-ca-cert\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.623619 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.623595    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-podinfo\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.625477 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.625455    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-envoy\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.681962 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.681891    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7pts\" (UniqueName: \"kubernetes.io/projected/610101d0-cf94-41b6-a21e-18e652eb9b03-kube-api-access-m7pts\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.682100 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.682054    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-token\") pod \"e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") " pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:25.692965 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:25.692938    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:26.162848 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.162814    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"]
Nov 03 21:25:26.202107 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:25:26.202080    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610101d0_cf94_41b6_a21e_18e652eb9b03.slice/crio-e1bcf553e7457c4909f8fee1cbad3f6c308ead0072043b740eaac5bdeb786042 WatchSource:0}: Error finding container e1bcf553e7457c4909f8fee1cbad3f6c308ead0072043b740eaac5bdeb786042: Status 404 returned error can't find the container with id e1bcf553e7457c4909f8fee1cbad3f6c308ead0072043b740eaac5bdeb786042
Nov 03 21:25:26.204312 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.204277    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:25:26.204399 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.204351    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:25:26.204399 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.204378    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:25:26.269303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.268754    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"]
Nov 03 21:25:26.272817 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.272795    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.332399 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.332321    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"]
Nov 03 21:25:26.381948 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.381901    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e7203b81-gw-openshift-default-dockercfg-9rwdf\""
Nov 03 21:25:26.437075 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.436983    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-podinfo\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.437075 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.437032    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istiod-ca-cert\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.437075 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.437068    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-workload-certs\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.437404 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.437261    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b85dz\" (UniqueName: \"kubernetes.io/projected/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-kube-api-access-b85dz\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.437404 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.437325    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-workload-socket\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.437511 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.437426    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-token\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.437511 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.437460    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-data\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.437511 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.437492    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-envoy\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.437647 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.437556    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-credential-socket\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.538876 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.538835    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-workload-certs\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.539053 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.538887    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b85dz\" (UniqueName: \"kubernetes.io/projected/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-kube-api-access-b85dz\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.539053 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.538925    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-workload-socket\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.539053 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.538961    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-token\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.539053 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.538987    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-data\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.539053 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.539015    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-envoy\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.539053 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.539043    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-credential-socket\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.539396 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.539084    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-podinfo\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.539396 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.539134    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istiod-ca-cert\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.539396 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.539332    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-workload-certs\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.539396 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.539350    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-workload-socket\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.539593 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.539572    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-credential-socket\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.539638 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.539624    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-data\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.540144 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.540123    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istiod-ca-cert\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.541594 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.541570    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-envoy\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.541939 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.541915    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-podinfo\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.595857 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.595825    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-token\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.596073 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.596053    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b85dz\" (UniqueName: \"kubernetes.io/projected/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-kube-api-access-b85dz\") pod \"e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") " pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.887493 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.887426    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:26.957074 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.945293    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph" event={"ID":"610101d0-cf94-41b6-a21e-18e652eb9b03","Type":"ContainerStarted","Data":"288fdff87ee272d32b860033ef32db18a1ff944e5830b97c229ae91d67455d9f"}
Nov 03 21:25:26.957074 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:26.945388    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph" event={"ID":"610101d0-cf94-41b6-a21e-18e652eb9b03","Type":"ContainerStarted","Data":"e1bcf553e7457c4909f8fee1cbad3f6c308ead0072043b740eaac5bdeb786042"}
Nov 03 21:25:27.068989 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:27.068909    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph" podStartSLOduration=2.068887545 podStartE2EDuration="2.068887545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:27.064891016 +0000 UTC m=+641.558812890" watchObservedRunningTime="2026-11-03 21:25:27.068887545 +0000 UTC m=+641.562809422"
Nov 03 21:25:27.311700 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:27.311672    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"]
Nov 03 21:25:27.312588 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:25:27.312562    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0013fad5_118c_4e7b_b51f_6fdb78d47bb4.slice/crio-15764b73b435eb13d2449ed5a06fe4200c862ff55cec8a054334a1796865ce4c WatchSource:0}: Error finding container 15764b73b435eb13d2449ed5a06fe4200c862ff55cec8a054334a1796865ce4c: Status 404 returned error can't find the container with id 15764b73b435eb13d2449ed5a06fe4200c862ff55cec8a054334a1796865ce4c
Nov 03 21:25:27.315056 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:27.315009    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:25:27.315178 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:27.315086    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:25:27.315178 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:27.315121    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:25:27.693652 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:27.693620    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:27.698738 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:27.698709    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:27.951929 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:27.951833    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td" event={"ID":"0013fad5-118c-4e7b-b51f-6fdb78d47bb4","Type":"ContainerStarted","Data":"0d93dd01f2687ba7b43c249262e903018833ca5f5e9972b53d2d9825f13e4119"}
Nov 03 21:25:27.951929 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:27.951901    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td" event={"ID":"0013fad5-118c-4e7b-b51f-6fdb78d47bb4","Type":"ContainerStarted","Data":"15764b73b435eb13d2449ed5a06fe4200c862ff55cec8a054334a1796865ce4c"}
Nov 03 21:25:27.951929 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:27.951930    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:27.952856 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:27.952832    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:25:27.991488 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:27.991462    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"]
Nov 03 21:25:27.995181 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:27.995163    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.051053 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.051023    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c2e645a4-openshift-default-dockercfg-lkhwg\""
Nov 03 21:25:28.055011 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.054980    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"]
Nov 03 21:25:28.070942 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.070881    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td" podStartSLOduration=2.070861636 podStartE2EDuration="2.070861636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:28.067539293 +0000 UTC m=+642.561461171" watchObservedRunningTime="2026-11-03 21:25:28.070861636 +0000 UTC m=+642.564783514"
Nov 03 21:25:28.157064 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.157030    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-token\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.157064 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.157073    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-credential-socket\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.157361 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.157101    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r49xb\" (UniqueName: \"kubernetes.io/projected/a253cf6d-2f11-4dcb-b788-116090cdfe8b-kube-api-access-r49xb\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.157361 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.157265    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-workload-socket\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.157361 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.157348    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-workload-certs\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.157518 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.157424    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-data\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.157518 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.157458    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-podinfo\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.157518 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.157506    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-envoy\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.157667 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.157543    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istiod-ca-cert\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.258584 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.258493    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-envoy\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.258584 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.258544    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istiod-ca-cert\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.258808 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.258608    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-token\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.258808 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.258646    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-credential-socket\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.258808 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.258675    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r49xb\" (UniqueName: \"kubernetes.io/projected/a253cf6d-2f11-4dcb-b788-116090cdfe8b-kube-api-access-r49xb\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.258808 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.258711    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-workload-socket\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.259025 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.258907    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-workload-certs\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.259025 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.258992    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-data\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.259129 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.259026    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-podinfo\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.259129 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.259061    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-workload-socket\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.259311 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.259286    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-workload-certs\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.259390 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.259367    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-data\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.259445 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.259403    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istiod-ca-cert\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.259528 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.259499    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-credential-socket\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.261328 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.261307    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-envoy\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.261787 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.261765    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-podinfo\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.315316 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.315287    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-token\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.315507 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.315487    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r49xb\" (UniqueName: \"kubernetes.io/projected/a253cf6d-2f11-4dcb-b788-116090cdfe8b-kube-api-access-r49xb\") pod \"e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") " pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.608614 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.608535    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:28.888344 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.888259    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:28.893477 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.893458    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:28.957596 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.957572    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:28.958328 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:28.958311    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:25:29.063419 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:29.063388    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"]
Nov 03 21:25:29.066214 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:25:29.066190    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda253cf6d_2f11_4dcb_b788_116090cdfe8b.slice/crio-f9b25a3a19c6b0f299e44099b255e2d2a056a09c2311896048f272bd3c78ca64 WatchSource:0}: Error finding container f9b25a3a19c6b0f299e44099b255e2d2a056a09c2311896048f272bd3c78ca64: Status 404 returned error can't find the container with id f9b25a3a19c6b0f299e44099b255e2d2a056a09c2311896048f272bd3c78ca64
Nov 03 21:25:29.068712 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:29.068683    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:25:29.068793 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:29.068743    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:25:29.068793 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:29.068769    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:25:29.965428 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:29.964985    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd" event={"ID":"a253cf6d-2f11-4dcb-b788-116090cdfe8b","Type":"ContainerStarted","Data":"07f2b805c892b58e80dd721b3defe9a14e67ce3d68ed4d7ed21fc72cb286eace"}
Nov 03 21:25:29.965428 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:29.965048    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd" event={"ID":"a253cf6d-2f11-4dcb-b788-116090cdfe8b","Type":"ContainerStarted","Data":"f9b25a3a19c6b0f299e44099b255e2d2a056a09c2311896048f272bd3c78ca64"}
Nov 03 21:25:30.190254 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:30.190196    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd" podStartSLOduration=3.1901813900000002 podStartE2EDuration="3.19018139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:30.138758386 +0000 UTC m=+644.632680275" watchObservedRunningTime="2026-11-03 21:25:30.19018139 +0000 UTC m=+644.684103264"
Nov 03 21:25:30.610250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:30.610208    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:30.615341 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:30.615315    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:30.971377 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:30.971349    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:30.972196 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:30.972177    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:32.745597 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:32.745560    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7"]
Nov 03 21:25:32.749730 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:32.749705    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7"
Nov 03 21:25:32.805827 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:32.805799    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7"]
Nov 03 21:25:32.806348 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:32.806323    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnjsm\" (UniqueName: \"kubernetes.io/projected/42e8cb6e-820e-45d6-8af3-fa981f2bfa7a-kube-api-access-tnjsm\") pod \"payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7\" (UID: \"42e8cb6e-820e-45d6-8af3-fa981f2bfa7a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7"
Nov 03 21:25:32.806538 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:32.806520    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e8cb6e-820e-45d6-8af3-fa981f2bfa7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7\" (UID: \"42e8cb6e-820e-45d6-8af3-fa981f2bfa7a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7"
Nov 03 21:25:32.852823 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:32.852785    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-35257bc0-dockercfg-mbjgv\""
Nov 03 21:25:32.852823 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:32.852823    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-35257bc0\""
Nov 03 21:25:32.907625 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:32.907575    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e8cb6e-820e-45d6-8af3-fa981f2bfa7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7\" (UID: \"42e8cb6e-820e-45d6-8af3-fa981f2bfa7a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7"
Nov 03 21:25:32.908018 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:32.907990    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnjsm\" (UniqueName: \"kubernetes.io/projected/42e8cb6e-820e-45d6-8af3-fa981f2bfa7a-kube-api-access-tnjsm\") pod \"payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7\" (UID: \"42e8cb6e-820e-45d6-8af3-fa981f2bfa7a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7"
Nov 03 21:25:32.909485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:32.909462    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e8cb6e-820e-45d6-8af3-fa981f2bfa7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7\" (UID: \"42e8cb6e-820e-45d6-8af3-fa981f2bfa7a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7"
Nov 03 21:25:32.965948 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:32.965908    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnjsm\" (UniqueName: \"kubernetes.io/projected/42e8cb6e-820e-45d6-8af3-fa981f2bfa7a-kube-api-access-tnjsm\") pod \"payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7\" (UID: \"42e8cb6e-820e-45d6-8af3-fa981f2bfa7a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7"
Nov 03 21:25:33.064365 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:33.064275    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7"
Nov 03 21:25:33.444571 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:33.444547    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7"]
Nov 03 21:25:33.446359 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:25:33.446328    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e8cb6e_820e_45d6_8af3_fa981f2bfa7a.slice/crio-b7c510734ba33050bcca20e170f8d581d389c34b08397d13bf9b2571c8f206ff WatchSource:0}: Error finding container b7c510734ba33050bcca20e170f8d581d389c34b08397d13bf9b2571c8f206ff: Status 404 returned error can't find the container with id b7c510734ba33050bcca20e170f8d581d389c34b08397d13bf9b2571c8f206ff
Nov 03 21:25:33.448423 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:33.448401    5053 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:25:33.988442 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:33.988407    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7" event={"ID":"42e8cb6e-820e-45d6-8af3-fa981f2bfa7a","Type":"ContainerStarted","Data":"b7c510734ba33050bcca20e170f8d581d389c34b08397d13bf9b2571c8f206ff"}
Nov 03 21:25:37.006559 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:37.006517    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7" event={"ID":"42e8cb6e-820e-45d6-8af3-fa981f2bfa7a","Type":"ContainerStarted","Data":"ddebab7a55a532c0ee1a1b4712bbc6ce4445b253d308d3950b84e7e64b2a1a17"}
Nov 03 21:25:37.006995 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:37.006680    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7"
Nov 03 21:25:37.120984 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:37.120927    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7" podStartSLOduration=2.226820988 podStartE2EDuration="5.120912227s" totalImagesPullingTime="2.894091239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:37.118919738 +0000 UTC m=+651.612841614" watchObservedRunningTime="2026-11-03 21:25:37.120912227 +0000 UTC m=+651.614834100"
Nov 03 21:25:37.672050 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:37.672021    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x"]
Nov 03 21:25:37.675777 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:37.675751    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x"
Nov 03 21:25:37.729875 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:37.729814    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-889495a7-dockercfg-qtbbp\""
Nov 03 21:25:37.729875 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:37.729813    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-889495a7\""
Nov 03 21:25:37.784841 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:37.783530    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x"]
Nov 03 21:25:37.868666 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:37.868626    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c4b9df-d9fe-4df9-9c03-94049a3e79ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x\" (UID: \"30c4b9df-d9fe-4df9-9c03-94049a3e79ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x"
Nov 03 21:25:37.868666 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:37.868660    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84sb9\" (UniqueName: \"kubernetes.io/projected/30c4b9df-d9fe-4df9-9c03-94049a3e79ce-kube-api-access-84sb9\") pod \"payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x\" (UID: \"30c4b9df-d9fe-4df9-9c03-94049a3e79ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x"
Nov 03 21:25:37.969466 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:37.969426    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c4b9df-d9fe-4df9-9c03-94049a3e79ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x\" (UID: \"30c4b9df-d9fe-4df9-9c03-94049a3e79ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x"
Nov 03 21:25:37.969647 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:37.969473    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84sb9\" (UniqueName: \"kubernetes.io/projected/30c4b9df-d9fe-4df9-9c03-94049a3e79ce-kube-api-access-84sb9\") pod \"payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x\" (UID: \"30c4b9df-d9fe-4df9-9c03-94049a3e79ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x"
Nov 03 21:25:37.970161 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:37.970142    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c4b9df-d9fe-4df9-9c03-94049a3e79ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x\" (UID: \"30c4b9df-d9fe-4df9-9c03-94049a3e79ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x"
Nov 03 21:25:38.026293 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:38.026260    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84sb9\" (UniqueName: \"kubernetes.io/projected/30c4b9df-d9fe-4df9-9c03-94049a3e79ce-kube-api-access-84sb9\") pod \"payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x\" (UID: \"30c4b9df-d9fe-4df9-9c03-94049a3e79ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x"
Nov 03 21:25:38.289840 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:38.289768    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x"
Nov 03 21:25:38.662912 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:38.662886    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x"]
Nov 03 21:25:38.664731 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:25:38.664703    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c4b9df_d9fe_4df9_9c03_94049a3e79ce.slice/crio-aefcf41cea8758d5a1f0779eb14818ac7acd55ea43f6d6f53f0286723d0a623a WatchSource:0}: Error finding container aefcf41cea8758d5a1f0779eb14818ac7acd55ea43f6d6f53f0286723d0a623a: Status 404 returned error can't find the container with id aefcf41cea8758d5a1f0779eb14818ac7acd55ea43f6d6f53f0286723d0a623a
Nov 03 21:25:39.019563 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:39.019521    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x" event={"ID":"30c4b9df-d9fe-4df9-9c03-94049a3e79ce","Type":"ContainerStarted","Data":"7f178403ffcf1491e19c50a3db32854f2976cb00c0a83dbf51f27c83cb665ba6"}
Nov 03 21:25:39.019750 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:39.019586    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x" event={"ID":"30c4b9df-d9fe-4df9-9c03-94049a3e79ce","Type":"ContainerStarted","Data":"aefcf41cea8758d5a1f0779eb14818ac7acd55ea43f6d6f53f0286723d0a623a"}
Nov 03 21:25:39.019750 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:39.019717    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x"
Nov 03 21:25:39.134654 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:39.134599    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x" podStartSLOduration=2.134583285 podStartE2EDuration="2.134583285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:39.133635563 +0000 UTC m=+653.627557463" watchObservedRunningTime="2026-11-03 21:25:39.134583285 +0000 UTC m=+653.628505160"
Nov 03 21:25:40.157522 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:40.157490    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc"]
Nov 03 21:25:40.161357 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:40.161332    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc"
Nov 03 21:25:40.191347 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:40.191313    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnvtf\" (UniqueName: \"kubernetes.io/projected/d6ffe32b-429e-4e1a-bdf3-6e30f287f6df-kube-api-access-tnvtf\") pod \"payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc\" (UID: \"d6ffe32b-429e-4e1a-bdf3-6e30f287f6df\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc"
Nov 03 21:25:40.191499 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:40.191482    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6ffe32b-429e-4e1a-bdf3-6e30f287f6df-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc\" (UID: \"d6ffe32b-429e-4e1a-bdf3-6e30f287f6df\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc"
Nov 03 21:25:40.264392 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:40.264350    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-e7203b81-dockercfg-hp422\""
Nov 03 21:25:40.264557 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:40.264436    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-e7203b81\""
Nov 03 21:25:40.266919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:40.266892    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc"]
Nov 03 21:25:40.292712 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:40.292675    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6ffe32b-429e-4e1a-bdf3-6e30f287f6df-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc\" (UID: \"d6ffe32b-429e-4e1a-bdf3-6e30f287f6df\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc"
Nov 03 21:25:40.292857 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:40.292750    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnvtf\" (UniqueName: \"kubernetes.io/projected/d6ffe32b-429e-4e1a-bdf3-6e30f287f6df-kube-api-access-tnvtf\") pod \"payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc\" (UID: \"d6ffe32b-429e-4e1a-bdf3-6e30f287f6df\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc"
Nov 03 21:25:40.293428 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:40.293407    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6ffe32b-429e-4e1a-bdf3-6e30f287f6df-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc\" (UID: \"d6ffe32b-429e-4e1a-bdf3-6e30f287f6df\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc"
Nov 03 21:25:40.316191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:40.316166    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw"]
Nov 03 21:25:40.320260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:40.320214    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw"
Nov 03 21:25:40.351879 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:40.351858    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnvtf\" (UniqueName: \"kubernetes.io/projected/d6ffe32b-429e-4e1a-bdf3-6e30f287f6df-kube-api-access-tnvtf\") pod \"payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc\" (UID: \"d6ffe32b-429e-4e1a-bdf3-6e30f287f6df\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc"
Nov 03 21:25:40.364736 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:40.364707    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw"]
Nov 03 21:25:40.393709 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:40.393678    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17334f9f-d4ee-4e20-85d8-f211ed2d6c62-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw\" (UID: \"17334f9f-d4ee-4e20-85d8-f211ed2d6c62\") " pod="openshift-ingress/payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw"
Nov 03 21:25:40.393885 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:40.393728    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmqbz\" (UniqueName: \"kubernetes.io/projected/17334f9f-d4ee-4e20-85d8-f211ed2d6c62-kube-api-access-bmqbz\") pod \"payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw\" (UID: \"17334f9f-d4ee-4e20-85d8-f211ed2d6c62\") " pod="openshift-ingress/payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw"
Nov 03 21:25:40.475711 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:40.475680    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc"
Nov 03 21:25:40.495465 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:40.495441    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17334f9f-d4ee-4e20-85d8-f211ed2d6c62-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw\" (UID: \"17334f9f-d4ee-4e20-85d8-f211ed2d6c62\") " pod="openshift-ingress/payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw"
Nov 03 21:25:40.495606 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:40.495487    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmqbz\" (UniqueName: \"kubernetes.io/projected/17334f9f-d4ee-4e20-85d8-f211ed2d6c62-kube-api-access-bmqbz\") pod \"payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw\" (UID: \"17334f9f-d4ee-4e20-85d8-f211ed2d6c62\") " pod="openshift-ingress/payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw"
Nov 03 21:25:40.496146 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:40.496126    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17334f9f-d4ee-4e20-85d8-f211ed2d6c62-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw\" (UID: \"17334f9f-d4ee-4e20-85d8-f211ed2d6c62\") " pod="openshift-ingress/payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw"
Nov 03 21:25:40.567578 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:40.560756    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmqbz\" (UniqueName: \"kubernetes.io/projected/17334f9f-d4ee-4e20-85d8-f211ed2d6c62-kube-api-access-bmqbz\") pod \"payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw\" (UID: \"17334f9f-d4ee-4e20-85d8-f211ed2d6c62\") " pod="openshift-ingress/payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw"
Nov 03 21:25:40.633010 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:40.632978    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw"
Nov 03 21:25:40.853709 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:40.853687    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc"]
Nov 03 21:25:40.855599 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:25:40.855575    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ffe32b_429e_4e1a_bdf3_6e30f287f6df.slice/crio-0a3bd6d4f9c1ed10aa7f942144baf334ee69ca731dcc6b7cd441af0999f19ba4 WatchSource:0}: Error finding container 0a3bd6d4f9c1ed10aa7f942144baf334ee69ca731dcc6b7cd441af0999f19ba4: Status 404 returned error can't find the container with id 0a3bd6d4f9c1ed10aa7f942144baf334ee69ca731dcc6b7cd441af0999f19ba4
Nov 03 21:25:40.932309 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:25:40.932279    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:25:41.008644 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:41.008615    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw"]
Nov 03 21:25:41.012710 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:25:41.012680    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17334f9f_d4ee_4e20_85d8_f211ed2d6c62.slice/crio-69d4b7dd6dbb863f34fd3486cf7438ef515e41de5266f4b7e8499053690657c1 WatchSource:0}: Error finding container 69d4b7dd6dbb863f34fd3486cf7438ef515e41de5266f4b7e8499053690657c1: Status 404 returned error can't find the container with id 69d4b7dd6dbb863f34fd3486cf7438ef515e41de5266f4b7e8499053690657c1
Nov 03 21:25:41.040919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:41.040886    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw" event={"ID":"17334f9f-d4ee-4e20-85d8-f211ed2d6c62","Type":"ContainerStarted","Data":"69d4b7dd6dbb863f34fd3486cf7438ef515e41de5266f4b7e8499053690657c1"}
Nov 03 21:25:41.042584 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:41.042547    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc" event={"ID":"d6ffe32b-429e-4e1a-bdf3-6e30f287f6df","Type":"ContainerStarted","Data":"cd50c2cd445015fa79bc8405518449201ce84b4c49aefd9bebbc436bff3c54cf"}
Nov 03 21:25:41.042713 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:41.042599    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc" event={"ID":"d6ffe32b-429e-4e1a-bdf3-6e30f287f6df","Type":"ContainerStarted","Data":"0a3bd6d4f9c1ed10aa7f942144baf334ee69ca731dcc6b7cd441af0999f19ba4"}
Nov 03 21:25:41.042932 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:41.042810    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc"
Nov 03 21:25:41.156515 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:41.156411    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc" podStartSLOduration=1.156396958 podStartE2EDuration="1.156396958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:41.155215276 +0000 UTC m=+655.649137153" watchObservedRunningTime="2026-11-03 21:25:41.156396958 +0000 UTC m=+655.650318832"
Nov 03 21:25:42.049047 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:42.049007    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw" event={"ID":"17334f9f-d4ee-4e20-85d8-f211ed2d6c62","Type":"ContainerStarted","Data":"4d83383e87d16da6ae10c0ff92ebcc63e7ecdbd187657a7176403ec40abe256f"}
Nov 03 21:25:42.165284 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:42.165215    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw" podStartSLOduration=2.165200702 podStartE2EDuration="2.165200702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:42.162192306 +0000 UTC m=+656.656114182" watchObservedRunningTime="2026-11-03 21:25:42.165200702 +0000 UTC m=+656.659122575"
Nov 03 21:25:42.718310 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:42.718279    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz"]
Nov 03 21:25:42.722448 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:42.722423    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz"
Nov 03 21:25:42.781480 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:42.779609    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz"]
Nov 03 21:25:42.781480 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:42.779791    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c2e645a4\""
Nov 03 21:25:42.821099 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:42.821059    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f9340d0-88c5-4a7a-a13d-2e28dd451ff8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz\" (UID: \"7f9340d0-88c5-4a7a-a13d-2e28dd451ff8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz"
Nov 03 21:25:42.821338 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:42.821220    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88pl2\" (UniqueName: \"kubernetes.io/projected/7f9340d0-88c5-4a7a-a13d-2e28dd451ff8-kube-api-access-88pl2\") pod \"payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz\" (UID: \"7f9340d0-88c5-4a7a-a13d-2e28dd451ff8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz"
Nov 03 21:25:42.824737 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:42.824714    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c2e645a4-dockercfg-v26bh\""
Nov 03 21:25:42.864084 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:42.864053    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr"]
Nov 03 21:25:42.867897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:42.867877    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr"
Nov 03 21:25:42.922891 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:42.922848    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f9340d0-88c5-4a7a-a13d-2e28dd451ff8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz\" (UID: \"7f9340d0-88c5-4a7a-a13d-2e28dd451ff8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz"
Nov 03 21:25:42.923067 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:42.922918    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa41bfe-320e-4171-abcc-db532e439c78-plugins-config-volume\") pod \"payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr\" (UID: \"dfa41bfe-320e-4171-abcc-db532e439c78\") " pod="openshift-ingress/payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr"
Nov 03 21:25:42.923067 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:42.922989    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887c2\" (UniqueName: \"kubernetes.io/projected/dfa41bfe-320e-4171-abcc-db532e439c78-kube-api-access-887c2\") pod \"payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr\" (UID: \"dfa41bfe-320e-4171-abcc-db532e439c78\") " pod="openshift-ingress/payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr"
Nov 03 21:25:42.923199 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:42.923107    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88pl2\" (UniqueName: \"kubernetes.io/projected/7f9340d0-88c5-4a7a-a13d-2e28dd451ff8-kube-api-access-88pl2\") pod \"payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz\" (UID: \"7f9340d0-88c5-4a7a-a13d-2e28dd451ff8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz"
Nov 03 21:25:42.924092 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:42.923970    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f9340d0-88c5-4a7a-a13d-2e28dd451ff8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz\" (UID: \"7f9340d0-88c5-4a7a-a13d-2e28dd451ff8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz"
Nov 03 21:25:42.925682 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:42.925658    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr"]
Nov 03 21:25:42.979675 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:42.979599    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88pl2\" (UniqueName: \"kubernetes.io/projected/7f9340d0-88c5-4a7a-a13d-2e28dd451ff8-kube-api-access-88pl2\") pod \"payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz\" (UID: \"7f9340d0-88c5-4a7a-a13d-2e28dd451ff8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz"
Nov 03 21:25:43.024214 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:43.024183    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa41bfe-320e-4171-abcc-db532e439c78-plugins-config-volume\") pod \"payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr\" (UID: \"dfa41bfe-320e-4171-abcc-db532e439c78\") " pod="openshift-ingress/payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr"
Nov 03 21:25:43.024400 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:43.024267    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887c2\" (UniqueName: \"kubernetes.io/projected/dfa41bfe-320e-4171-abcc-db532e439c78-kube-api-access-887c2\") pod \"payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr\" (UID: \"dfa41bfe-320e-4171-abcc-db532e439c78\") " pod="openshift-ingress/payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr"
Nov 03 21:25:43.024837 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:43.024817    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa41bfe-320e-4171-abcc-db532e439c78-plugins-config-volume\") pod \"payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr\" (UID: \"dfa41bfe-320e-4171-abcc-db532e439c78\") " pod="openshift-ingress/payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr"
Nov 03 21:25:43.036226 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:43.036203    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz"
Nov 03 21:25:43.055481 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:43.055108    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw"
Nov 03 21:25:43.080747 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:43.080722    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-887c2\" (UniqueName: \"kubernetes.io/projected/dfa41bfe-320e-4171-abcc-db532e439c78-kube-api-access-887c2\") pod \"payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr\" (UID: \"dfa41bfe-320e-4171-abcc-db532e439c78\") " pod="openshift-ingress/payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr"
Nov 03 21:25:43.181303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:43.181267    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr"
Nov 03 21:25:43.406335 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:43.406312    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz"]
Nov 03 21:25:43.407686 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:25:43.407661    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f9340d0_88c5_4a7a_a13d_2e28dd451ff8.slice/crio-daaf9ed5fe044fcda84bd64d4f294be632ccbe6352d479c0c5cd443918ff0336 WatchSource:0}: Error finding container daaf9ed5fe044fcda84bd64d4f294be632ccbe6352d479c0c5cd443918ff0336: Status 404 returned error can't find the container with id daaf9ed5fe044fcda84bd64d4f294be632ccbe6352d479c0c5cd443918ff0336
Nov 03 21:25:43.562125 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:43.562098    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr"]
Nov 03 21:25:43.563349 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:25:43.563322    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa41bfe_320e_4171_abcc_db532e439c78.slice/crio-16359ef5e2e90465c96aa07a6f462724f387513c4db5b425761367131a5c9269 WatchSource:0}: Error finding container 16359ef5e2e90465c96aa07a6f462724f387513c4db5b425761367131a5c9269: Status 404 returned error can't find the container with id 16359ef5e2e90465c96aa07a6f462724f387513c4db5b425761367131a5c9269
Nov 03 21:25:44.061846 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:44.061813    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr" event={"ID":"dfa41bfe-320e-4171-abcc-db532e439c78","Type":"ContainerStarted","Data":"6f2c56732d10264a5f0d88cd73d26930a89c1b7809f022fa0da8e823676fb983"}
Nov 03 21:25:44.062310 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:44.061871    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr" event={"ID":"dfa41bfe-320e-4171-abcc-db532e439c78","Type":"ContainerStarted","Data":"16359ef5e2e90465c96aa07a6f462724f387513c4db5b425761367131a5c9269"}
Nov 03 21:25:44.062310 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:44.061897    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr"
Nov 03 21:25:44.063450 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:44.063426    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz" event={"ID":"7f9340d0-88c5-4a7a-a13d-2e28dd451ff8","Type":"ContainerStarted","Data":"32fc6d7632d84e2e733ee25b6069b1b61246c0d14bfa249091cd1b1a0e39f85a"}
Nov 03 21:25:44.063549 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:44.063468    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz" event={"ID":"7f9340d0-88c5-4a7a-a13d-2e28dd451ff8","Type":"ContainerStarted","Data":"daaf9ed5fe044fcda84bd64d4f294be632ccbe6352d479c0c5cd443918ff0336"}
Nov 03 21:25:44.063590 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:44.063542    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz"
Nov 03 21:25:44.174867 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:44.174810    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr" podStartSLOduration=2.174793155 podStartE2EDuration="2.174793155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:44.174467919 +0000 UTC m=+658.668389794" watchObservedRunningTime="2026-11-03 21:25:44.174793155 +0000 UTC m=+658.668715031"
Nov 03 21:25:44.292298 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:44.289422    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz" podStartSLOduration=2.289400048 podStartE2EDuration="2.289400048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:44.287497617 +0000 UTC m=+658.781419493" watchObservedRunningTime="2026-11-03 21:25:44.289400048 +0000 UTC m=+658.783321925"
Nov 03 21:25:48.039263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:48.038487    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7"
Nov 03 21:25:49.871283 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:49.871218    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc"]
Nov 03 21:25:49.876212 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:49.876195    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc"
Nov 03 21:25:49.885457 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:49.885432    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbdf9\" (UniqueName: \"kubernetes.io/projected/6d4a1b84-b85a-4ff9-b2ab-31d651df7059-kube-api-access-mbdf9\") pod \"payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc\" (UID: \"6d4a1b84-b85a-4ff9-b2ab-31d651df7059\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc"
Nov 03 21:25:49.885589 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:49.885496    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d4a1b84-b85a-4ff9-b2ab-31d651df7059-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc\" (UID: \"6d4a1b84-b85a-4ff9-b2ab-31d651df7059\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc"
Nov 03 21:25:49.932063 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:49.932028    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc"]
Nov 03 21:25:49.983293 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:49.982529    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-daf21c29\""
Nov 03 21:25:49.983293 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:49.982662    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-daf21c29-dockercfg-rfncg\""
Nov 03 21:25:49.986165 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:49.986134    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d4a1b84-b85a-4ff9-b2ab-31d651df7059-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc\" (UID: \"6d4a1b84-b85a-4ff9-b2ab-31d651df7059\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc"
Nov 03 21:25:49.986295 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:49.986278    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbdf9\" (UniqueName: \"kubernetes.io/projected/6d4a1b84-b85a-4ff9-b2ab-31d651df7059-kube-api-access-mbdf9\") pod \"payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc\" (UID: \"6d4a1b84-b85a-4ff9-b2ab-31d651df7059\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc"
Nov 03 21:25:49.987027 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:49.987011    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d4a1b84-b85a-4ff9-b2ab-31d651df7059-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc\" (UID: \"6d4a1b84-b85a-4ff9-b2ab-31d651df7059\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc"
Nov 03 21:25:50.034614 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:50.034578    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx"]
Nov 03 21:25:50.039386 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:50.039366    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x"
Nov 03 21:25:50.039511 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:50.039444    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx"
Nov 03 21:25:50.046969 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:50.046944    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbdf9\" (UniqueName: \"kubernetes.io/projected/6d4a1b84-b85a-4ff9-b2ab-31d651df7059-kube-api-access-mbdf9\") pod \"payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc\" (UID: \"6d4a1b84-b85a-4ff9-b2ab-31d651df7059\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc"
Nov 03 21:25:50.087047 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:50.087012    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx"]
Nov 03 21:25:50.087220 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:50.087155    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rc4n\" (UniqueName: \"kubernetes.io/projected/8d5c416d-5668-4718-965e-e5a2b5e0ff5c-kube-api-access-2rc4n\") pod \"payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx\" (UID: \"8d5c416d-5668-4718-965e-e5a2b5e0ff5c\") " pod="openshift-ingress/payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx"
Nov 03 21:25:50.087304 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:50.087289    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5c416d-5668-4718-965e-e5a2b5e0ff5c-plugins-config-volume\") pod \"payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx\" (UID: \"8d5c416d-5668-4718-965e-e5a2b5e0ff5c\") " pod="openshift-ingress/payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx"
Nov 03 21:25:50.188691 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:50.188613    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rc4n\" (UniqueName: \"kubernetes.io/projected/8d5c416d-5668-4718-965e-e5a2b5e0ff5c-kube-api-access-2rc4n\") pod \"payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx\" (UID: \"8d5c416d-5668-4718-965e-e5a2b5e0ff5c\") " pod="openshift-ingress/payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx"
Nov 03 21:25:50.188691 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:50.188671    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5c416d-5668-4718-965e-e5a2b5e0ff5c-plugins-config-volume\") pod \"payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx\" (UID: \"8d5c416d-5668-4718-965e-e5a2b5e0ff5c\") " pod="openshift-ingress/payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx"
Nov 03 21:25:50.189154 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:50.189129    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc"
Nov 03 21:25:50.189427 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:50.189400    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5c416d-5668-4718-965e-e5a2b5e0ff5c-plugins-config-volume\") pod \"payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx\" (UID: \"8d5c416d-5668-4718-965e-e5a2b5e0ff5c\") " pod="openshift-ingress/payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx"
Nov 03 21:25:50.245397 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:50.245347    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rc4n\" (UniqueName: \"kubernetes.io/projected/8d5c416d-5668-4718-965e-e5a2b5e0ff5c-kube-api-access-2rc4n\") pod \"payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx\" (UID: \"8d5c416d-5668-4718-965e-e5a2b5e0ff5c\") " pod="openshift-ingress/payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx"
Nov 03 21:25:50.363849 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:50.363805    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx"
Nov 03 21:25:50.589874 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:50.589840    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc"]
Nov 03 21:25:50.591246 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:25:50.591209    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d4a1b84_b85a_4ff9_b2ab_31d651df7059.slice/crio-c1fc337b6758d6dee7cd74f33fbaa2463e2e69d39aa980c76b7a9dd086f5dcbf WatchSource:0}: Error finding container c1fc337b6758d6dee7cd74f33fbaa2463e2e69d39aa980c76b7a9dd086f5dcbf: Status 404 returned error can't find the container with id c1fc337b6758d6dee7cd74f33fbaa2463e2e69d39aa980c76b7a9dd086f5dcbf
Nov 03 21:25:50.773371 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:50.773348    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx"]
Nov 03 21:25:50.775433 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:25:50.775413    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5c416d_5668_4718_965e_e5a2b5e0ff5c.slice/crio-4bb088591bb80720c51441012cffa5a5b9d38defb5df7d575337c9b6da7970a8 WatchSource:0}: Error finding container 4bb088591bb80720c51441012cffa5a5b9d38defb5df7d575337c9b6da7970a8: Status 404 returned error can't find the container with id 4bb088591bb80720c51441012cffa5a5b9d38defb5df7d575337c9b6da7970a8
Nov 03 21:25:51.125329 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:51.124643    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx" event={"ID":"8d5c416d-5668-4718-965e-e5a2b5e0ff5c","Type":"ContainerStarted","Data":"6fcf3f9f1bf86cffb0ed0b894de4de3f8c199ebead73974d1ce90741de492409"}
Nov 03 21:25:51.125329 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:51.124714    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx" event={"ID":"8d5c416d-5668-4718-965e-e5a2b5e0ff5c","Type":"ContainerStarted","Data":"4bb088591bb80720c51441012cffa5a5b9d38defb5df7d575337c9b6da7970a8"}
Nov 03 21:25:51.125828 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:51.125368    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx"
Nov 03 21:25:51.127692 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:51.127663    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc" event={"ID":"6d4a1b84-b85a-4ff9-b2ab-31d651df7059","Type":"ContainerStarted","Data":"eb1125cac2ec0abc9253cecde3b7eceb9fde6a1a85306cc7d492073c31002509"}
Nov 03 21:25:51.127897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:51.127882    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc"
Nov 03 21:25:51.128029 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:51.128001    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc" event={"ID":"6d4a1b84-b85a-4ff9-b2ab-31d651df7059","Type":"ContainerStarted","Data":"c1fc337b6758d6dee7cd74f33fbaa2463e2e69d39aa980c76b7a9dd086f5dcbf"}
Nov 03 21:25:51.238822 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:51.238764    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx" podStartSLOduration=2.238746178 podStartE2EDuration="2.238746178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:51.238182933 +0000 UTC m=+665.732104807" watchObservedRunningTime="2026-11-03 21:25:51.238746178 +0000 UTC m=+665.732668053"
Nov 03 21:25:51.352660 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:51.352211    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc" podStartSLOduration=2.35219371 podStartE2EDuration="2.35219371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:51.350146546 +0000 UTC m=+665.844068422" watchObservedRunningTime="2026-11-03 21:25:51.35219371 +0000 UTC m=+665.846115583"
Nov 03 21:25:52.052405 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:52.052379    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc"
Nov 03 21:25:54.066756 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:54.066726    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw"
Nov 03 21:25:55.072713 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:55.072678    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr"
Nov 03 21:25:55.073111 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:55.072761    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz"
Nov 03 21:25:56.165182 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:56.165102    5053 generic.go:356] "Generic (PLEG): container finished" podID="a253cf6d-2f11-4dcb-b788-116090cdfe8b" containerID="07f2b805c892b58e80dd721b3defe9a14e67ce3d68ed4d7ed21fc72cb286eace" exitCode=0
Nov 03 21:25:56.165182 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:56.165171    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd" event={"ID":"a253cf6d-2f11-4dcb-b788-116090cdfe8b","Type":"ContainerDied","Data":"07f2b805c892b58e80dd721b3defe9a14e67ce3d68ed4d7ed21fc72cb286eace"}
Nov 03 21:25:56.165684 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:56.165578    5053 scope.go:122] "RemoveContainer" containerID="07f2b805c892b58e80dd721b3defe9a14e67ce3d68ed4d7ed21fc72cb286eace"
Nov 03 21:25:56.166871 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:56.166842    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:25:56.166939 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:56.166910    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:25:56.166980 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:56.166942    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:25:57.174947 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:57.174909    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd" event={"ID":"a253cf6d-2f11-4dcb-b788-116090cdfe8b","Type":"ContainerStarted","Data":"a6811d397649cfdb0dcf18b13814896cec308d96b151a517691e0194d8e5e97a"}
Nov 03 21:25:57.616230 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:57.616199    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:25:57.622303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:57.618638    5053 patch_prober.go:29] interesting pod/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 03 21:25:57.622303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:57.618690    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd" podUID="a253cf6d-2f11-4dcb-b788-116090cdfe8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 03 21:25:58.621267 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:58.611380    5053 patch_prober.go:29] interesting pod/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 03 21:25:58.621267 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:58.611446    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd" podUID="a253cf6d-2f11-4dcb-b788-116090cdfe8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 03 21:25:59.616265 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:59.612634    5053 patch_prober.go:29] interesting pod/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 03 21:25:59.616265 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:25:59.612706    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd" podUID="a253cf6d-2f11-4dcb-b788-116090cdfe8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 03 21:26:00.624692 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:00.624645    5053 patch_prober.go:29] interesting pod/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 03 21:26:00.625199 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:00.624719    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd" podUID="a253cf6d-2f11-4dcb-b788-116090cdfe8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 03 21:26:00.972008 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:00.971982    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:26:01.610723 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:01.610684    5053 patch_prober.go:29] interesting pod/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 03 21:26:01.611057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:01.611027    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd" podUID="a253cf6d-2f11-4dcb-b788-116090cdfe8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 03 21:26:02.137212 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:02.137183    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc"
Nov 03 21:26:02.137669 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:02.137287    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx"
Nov 03 21:26:02.613632 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:02.613593    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:26:02.614637 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:02.614617    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:26:06.236294 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:06.236262    5053 generic.go:356] "Generic (PLEG): container finished" podID="610101d0-cf94-41b6-a21e-18e652eb9b03" containerID="288fdff87ee272d32b860033ef32db18a1ff944e5830b97c229ae91d67455d9f" exitCode=0
Nov 03 21:26:06.236718 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:06.236329    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph" event={"ID":"610101d0-cf94-41b6-a21e-18e652eb9b03","Type":"ContainerDied","Data":"288fdff87ee272d32b860033ef32db18a1ff944e5830b97c229ae91d67455d9f"}
Nov 03 21:26:06.236718 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:06.236692    5053 scope.go:122] "RemoveContainer" containerID="288fdff87ee272d32b860033ef32db18a1ff944e5830b97c229ae91d67455d9f"
Nov 03 21:26:06.238178 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:06.238140    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:26:06.238295 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:06.238250    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:26:06.238341 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:06.238304    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:26:07.254917 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:07.254882    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph" event={"ID":"610101d0-cf94-41b6-a21e-18e652eb9b03","Type":"ContainerStarted","Data":"0eb784963ed15b7ac5e8c8e38aaaacba01fbee65657be7808a90290f90b1b648"}
Nov 03 21:26:07.693373 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:07.693349    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:26:07.693627 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:07.693575    5053 patch_prober.go:29] interesting pod/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 03 21:26:07.693671 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:07.693621    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph" podUID="610101d0-cf94-41b6-a21e-18e652eb9b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 03 21:26:08.693658 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:08.693622    5053 patch_prober.go:29] interesting pod/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 03 21:26:08.694062 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:08.693680    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph" podUID="610101d0-cf94-41b6-a21e-18e652eb9b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 03 21:26:09.694177 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:09.694145    5053 patch_prober.go:29] interesting pod/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 03 21:26:09.694737 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:09.694224    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph" podUID="610101d0-cf94-41b6-a21e-18e652eb9b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 03 21:26:10.694282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:10.694229    5053 patch_prober.go:29] interesting pod/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 03 21:26:10.694745 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:10.694308    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph" podUID="610101d0-cf94-41b6-a21e-18e652eb9b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 03 21:26:10.972408 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:26:10.972323    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:26:11.697267 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:11.694639    5053 patch_prober.go:29] interesting pod/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 03 21:26:11.697267 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:11.694698    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph" podUID="610101d0-cf94-41b6-a21e-18e652eb9b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 03 21:26:12.698118 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:12.698084    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:26:12.698616 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:12.698447    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:26:12.699364 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:12.699344    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:26:15.000405 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:15.000358    5053 patch_prober.go:29] interesting pod/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:26:15.000889 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:15.000438    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td" podUID="0013fad5-118c-4e7b-b51f-6fdb78d47bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:26:16.196469 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:16.196441    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-889495a7-bcpl4"]
Nov 03 21:26:16.199414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:16.199399    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-889495a7-bcpl4"
Nov 03 21:26:16.254716 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:16.254686    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:26:16.278497 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:16.278472    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ddf660-d21e-444b-8567-8de4b71d1227-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-889495a7-bcpl4\" (UID: \"96ddf660-d21e-444b-8567-8de4b71d1227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-889495a7-bcpl4"
Nov 03 21:26:16.278644 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:16.278507    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ddf660-d21e-444b-8567-8de4b71d1227-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-889495a7-bcpl4\" (UID: \"96ddf660-d21e-444b-8567-8de4b71d1227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-889495a7-bcpl4"
Nov 03 21:26:16.278644 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:16.278606    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb54r\" (UniqueName: \"kubernetes.io/projected/96ddf660-d21e-444b-8567-8de4b71d1227-kube-api-access-nb54r\") pod \"maas-api-revoke-keys-e2e-ait-889495a7-bcpl4\" (UID: \"96ddf660-d21e-444b-8567-8de4b71d1227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-889495a7-bcpl4"
Nov 03 21:26:16.304085 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:16.304062    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:26:16.304286 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:16.304165    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:26:16.304286 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:16.304166    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:26:16.306990 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:16.306970    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-889495a7-bcpl4"]
Nov 03 21:26:16.314330 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:16.314307    5053 generic.go:356] "Generic (PLEG): container finished" podID="0013fad5-118c-4e7b-b51f-6fdb78d47bb4" containerID="0d93dd01f2687ba7b43c249262e903018833ca5f5e9972b53d2d9825f13e4119" exitCode=0
Nov 03 21:26:16.314440 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:16.314379    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td" event={"ID":"0013fad5-118c-4e7b-b51f-6fdb78d47bb4","Type":"ContainerDied","Data":"0d93dd01f2687ba7b43c249262e903018833ca5f5e9972b53d2d9825f13e4119"}
Nov 03 21:26:16.314736 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:16.314723    5053 scope.go:122] "RemoveContainer" containerID="0d93dd01f2687ba7b43c249262e903018833ca5f5e9972b53d2d9825f13e4119"
Nov 03 21:26:16.315984 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:16.315955    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:26:16.316059 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:16.316021    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:26:16.316059 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:16.316053    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:26:16.379350 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:16.379325    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ddf660-d21e-444b-8567-8de4b71d1227-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-889495a7-bcpl4\" (UID: \"96ddf660-d21e-444b-8567-8de4b71d1227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-889495a7-bcpl4"
Nov 03 21:26:16.379464 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:16.379423    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb54r\" (UniqueName: \"kubernetes.io/projected/96ddf660-d21e-444b-8567-8de4b71d1227-kube-api-access-nb54r\") pod \"maas-api-revoke-keys-e2e-ait-889495a7-bcpl4\" (UID: \"96ddf660-d21e-444b-8567-8de4b71d1227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-889495a7-bcpl4"
Nov 03 21:26:16.379628 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:16.379606    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ddf660-d21e-444b-8567-8de4b71d1227-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-889495a7-bcpl4\" (UID: \"96ddf660-d21e-444b-8567-8de4b71d1227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-889495a7-bcpl4"
Nov 03 21:26:16.380055 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:16.380036    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ddf660-d21e-444b-8567-8de4b71d1227-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-889495a7-bcpl4\" (UID: \"96ddf660-d21e-444b-8567-8de4b71d1227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-889495a7-bcpl4"
Nov 03 21:26:16.381870 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:16.381846    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ddf660-d21e-444b-8567-8de4b71d1227-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-889495a7-bcpl4\" (UID: \"96ddf660-d21e-444b-8567-8de4b71d1227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-889495a7-bcpl4"
Nov 03 21:26:16.436463 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:16.436435    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb54r\" (UniqueName: \"kubernetes.io/projected/96ddf660-d21e-444b-8567-8de4b71d1227-kube-api-access-nb54r\") pod \"maas-api-revoke-keys-e2e-ait-889495a7-bcpl4\" (UID: \"96ddf660-d21e-444b-8567-8de4b71d1227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-889495a7-bcpl4"
Nov 03 21:26:16.511598 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:16.511519    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-889495a7-bcpl4"
Nov 03 21:26:16.895672 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:16.895641    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-889495a7-bcpl4"]
Nov 03 21:26:16.897023 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:26:16.896996    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ddf660_d21e_444b_8567_8de4b71d1227.slice/crio-db4301be64e531c4611fea997aceb132fd62c5fc112e0968fcd7c7c7652f5f89 WatchSource:0}: Error finding container db4301be64e531c4611fea997aceb132fd62c5fc112e0968fcd7c7c7652f5f89: Status 404 returned error can't find the container with id db4301be64e531c4611fea997aceb132fd62c5fc112e0968fcd7c7c7652f5f89
Nov 03 21:26:17.333598 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:17.333568    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td" event={"ID":"0013fad5-118c-4e7b-b51f-6fdb78d47bb4","Type":"ContainerStarted","Data":"bc65cfdacbd84cec9a6ca6d545cc1d151c447a0e853692de19931dec6f822430"}
Nov 03 21:26:17.334739 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:17.334718    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-889495a7-bcpl4" event={"ID":"96ddf660-d21e-444b-8567-8de4b71d1227","Type":"ContainerStarted","Data":"db4301be64e531c4611fea997aceb132fd62c5fc112e0968fcd7c7c7652f5f89"}
Nov 03 21:26:17.887822 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:17.887786    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:26:17.888019 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:17.887996    5053 patch_prober.go:29] interesting pod/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 03 21:26:17.888069 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:17.888048    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td" podUID="0013fad5-118c-4e7b-b51f-6fdb78d47bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 03 21:26:18.888117 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:18.888082    5053 patch_prober.go:29] interesting pod/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 03 21:26:18.888546 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:18.888137    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td" podUID="0013fad5-118c-4e7b-b51f-6fdb78d47bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 03 21:26:19.025652 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:19.025610    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58"]
Nov 03 21:26:19.029275 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:19.029223    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58"
Nov 03 21:26:19.067705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:19.067669    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58"]
Nov 03 21:26:19.140965 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:19.140872    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3759306a-f917-4d2d-98ee-0bdc38a55460-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58\" (UID: \"3759306a-f917-4d2d-98ee-0bdc38a55460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58"
Nov 03 21:26:19.140965 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:19.140934    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3759306a-f917-4d2d-98ee-0bdc38a55460-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58\" (UID: \"3759306a-f917-4d2d-98ee-0bdc38a55460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58"
Nov 03 21:26:19.141172 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:19.141034    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dhct\" (UniqueName: \"kubernetes.io/projected/3759306a-f917-4d2d-98ee-0bdc38a55460-kube-api-access-8dhct\") pod \"maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58\" (UID: \"3759306a-f917-4d2d-98ee-0bdc38a55460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58"
Nov 03 21:26:19.242357 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:19.242322    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3759306a-f917-4d2d-98ee-0bdc38a55460-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58\" (UID: \"3759306a-f917-4d2d-98ee-0bdc38a55460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58"
Nov 03 21:26:19.242635 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:19.242369    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3759306a-f917-4d2d-98ee-0bdc38a55460-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58\" (UID: \"3759306a-f917-4d2d-98ee-0bdc38a55460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58"
Nov 03 21:26:19.242635 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:19.242401    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dhct\" (UniqueName: \"kubernetes.io/projected/3759306a-f917-4d2d-98ee-0bdc38a55460-kube-api-access-8dhct\") pod \"maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58\" (UID: \"3759306a-f917-4d2d-98ee-0bdc38a55460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58"
Nov 03 21:26:19.243193 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:19.243157    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3759306a-f917-4d2d-98ee-0bdc38a55460-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58\" (UID: \"3759306a-f917-4d2d-98ee-0bdc38a55460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58"
Nov 03 21:26:19.245179 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:19.245155    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3759306a-f917-4d2d-98ee-0bdc38a55460-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58\" (UID: \"3759306a-f917-4d2d-98ee-0bdc38a55460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58"
Nov 03 21:26:19.299494 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:19.299459    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dhct\" (UniqueName: \"kubernetes.io/projected/3759306a-f917-4d2d-98ee-0bdc38a55460-kube-api-access-8dhct\") pod \"maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58\" (UID: \"3759306a-f917-4d2d-98ee-0bdc38a55460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58"
Nov 03 21:26:19.341705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:19.341668    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58"
Nov 03 21:26:19.756911 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:19.756863    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58"]
Nov 03 21:26:19.888816 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:19.888780    5053 patch_prober.go:29] interesting pod/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 03 21:26:19.889331 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:19.888845    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td" podUID="0013fad5-118c-4e7b-b51f-6fdb78d47bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 03 21:26:20.353726 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:20.353686    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58" event={"ID":"3759306a-f917-4d2d-98ee-0bdc38a55460","Type":"ContainerStarted","Data":"6504bf6a2c08d75d37baee5267380269c7e626af7a1bd41d35e121821e524ebc"}
Nov 03 21:26:20.888578 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:20.888514    5053 patch_prober.go:29] interesting pod/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 03 21:26:20.888578 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:20.888571    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td" podUID="0013fad5-118c-4e7b-b51f-6fdb78d47bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 03 21:26:21.888370 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:21.888334    5053 patch_prober.go:29] interesting pod/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 03 21:26:21.888765 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:21.888390    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td" podUID="0013fad5-118c-4e7b-b51f-6fdb78d47bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 03 21:26:22.367310 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:22.367262    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58" event={"ID":"3759306a-f917-4d2d-98ee-0bdc38a55460","Type":"ContainerStarted","Data":"c76669506c3d0997a536de5ebba7c56595ac80351ffd3fd517fa3ad9ff920bb6"}
Nov 03 21:26:22.368852 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:22.368827    5053 generic.go:356] "Generic (PLEG): container finished" podID="96ddf660-d21e-444b-8567-8de4b71d1227" containerID="0786710ff2ec8795d4510303da124d048d8fd58613256ff03ab0ff6d43590eb7" exitCode=0
Nov 03 21:26:22.368970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:22.368862    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-889495a7-bcpl4" event={"ID":"96ddf660-d21e-444b-8567-8de4b71d1227","Type":"ContainerDied","Data":"0786710ff2ec8795d4510303da124d048d8fd58613256ff03ab0ff6d43590eb7"}
Nov 03 21:26:22.479466 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:22.479362    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58" podStartSLOduration=2.415784931 podStartE2EDuration="4.479346024s" totalImagesPullingTime="2.063561093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:22.478826314 +0000 UTC m=+696.972748190" watchObservedRunningTime="2026-11-03 21:26:22.479346024 +0000 UTC m=+696.973267900"
Nov 03 21:26:22.892110 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:22.892075    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:26:22.892585 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:22.892509    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:26:22.893075 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:22.893055    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:26:23.375439 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:23.375406    5053 generic.go:356] "Generic (PLEG): container finished" podID="3759306a-f917-4d2d-98ee-0bdc38a55460" containerID="c76669506c3d0997a536de5ebba7c56595ac80351ffd3fd517fa3ad9ff920bb6" exitCode=0
Nov 03 21:26:23.375620 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:23.375469    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58" event={"ID":"3759306a-f917-4d2d-98ee-0bdc38a55460","Type":"ContainerDied","Data":"c76669506c3d0997a536de5ebba7c56595ac80351ffd3fd517fa3ad9ff920bb6"}
Nov 03 21:26:23.519128 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:23.519102    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-889495a7-bcpl4"
Nov 03 21:26:23.609838 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:23.609804    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96ddf660-d21e-444b-8567-8de4b71d1227-kube-api-access-nb54r\" (UniqueName: \"kubernetes.io/projected/96ddf660-d21e-444b-8567-8de4b71d1227-kube-api-access-nb54r\") pod \"96ddf660-d21e-444b-8567-8de4b71d1227\" (UID: \"96ddf660-d21e-444b-8567-8de4b71d1227\") "
Nov 03 21:26:23.610007 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:23.609926    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96ddf660-d21e-444b-8567-8de4b71d1227-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ddf660-d21e-444b-8567-8de4b71d1227-maas-api-xks-ca\") pod \"96ddf660-d21e-444b-8567-8de4b71d1227\" (UID: \"96ddf660-d21e-444b-8567-8de4b71d1227\") "
Nov 03 21:26:23.610057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:23.610023    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96ddf660-d21e-444b-8567-8de4b71d1227-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ddf660-d21e-444b-8567-8de4b71d1227-maas-api-openshift-ca\") pod \"96ddf660-d21e-444b-8567-8de4b71d1227\" (UID: \"96ddf660-d21e-444b-8567-8de4b71d1227\") "
Nov 03 21:26:23.610432 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:23.610406    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96ddf660-d21e-444b-8567-8de4b71d1227-maas-api-openshift-ca" pod "96ddf660-d21e-444b-8567-8de4b71d1227" (UID: "96ddf660-d21e-444b-8567-8de4b71d1227"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:23.612189 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:23.612166    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96ddf660-d21e-444b-8567-8de4b71d1227-maas-api-xks-ca" pod "96ddf660-d21e-444b-8567-8de4b71d1227" (UID: "96ddf660-d21e-444b-8567-8de4b71d1227"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:26:23.612309 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:23.612217    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ddf660-d21e-444b-8567-8de4b71d1227-kube-api-access-nb54r" pod "96ddf660-d21e-444b-8567-8de4b71d1227" (UID: "96ddf660-d21e-444b-8567-8de4b71d1227"). InnerVolumeSpecName "kube-api-access-nb54r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:23.711787 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:23.711756    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ddf660-d21e-444b-8567-8de4b71d1227-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:23.711787 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:23.711783    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb54r\" (UniqueName: \"kubernetes.io/projected/96ddf660-d21e-444b-8567-8de4b71d1227-kube-api-access-nb54r\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:23.711787 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:23.711792    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ddf660-d21e-444b-8567-8de4b71d1227-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:24.382029 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:24.381986    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-889495a7-bcpl4" event={"ID":"96ddf660-d21e-444b-8567-8de4b71d1227","Type":"ContainerDied","Data":"db4301be64e531c4611fea997aceb132fd62c5fc112e0968fcd7c7c7652f5f89"}
Nov 03 21:26:24.382476 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:24.382033    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-889495a7-bcpl4"
Nov 03 21:26:24.382476 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:24.382043    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db4301be64e531c4611fea997aceb132fd62c5fc112e0968fcd7c7c7652f5f89"
Nov 03 21:26:24.527943 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:24.527917    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58"
Nov 03 21:26:24.621628 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:24.621597    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3759306a-f917-4d2d-98ee-0bdc38a55460-kube-api-access-8dhct\" (UniqueName: \"kubernetes.io/projected/3759306a-f917-4d2d-98ee-0bdc38a55460-kube-api-access-8dhct\") pod \"3759306a-f917-4d2d-98ee-0bdc38a55460\" (UID: \"3759306a-f917-4d2d-98ee-0bdc38a55460\") "
Nov 03 21:26:24.621833 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:24.621706    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3759306a-f917-4d2d-98ee-0bdc38a55460-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3759306a-f917-4d2d-98ee-0bdc38a55460-maas-api-xks-ca\") pod \"3759306a-f917-4d2d-98ee-0bdc38a55460\" (UID: \"3759306a-f917-4d2d-98ee-0bdc38a55460\") "
Nov 03 21:26:24.621833 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:24.621735    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3759306a-f917-4d2d-98ee-0bdc38a55460-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3759306a-f917-4d2d-98ee-0bdc38a55460-maas-api-openshift-ca\") pod \"3759306a-f917-4d2d-98ee-0bdc38a55460\" (UID: \"3759306a-f917-4d2d-98ee-0bdc38a55460\") "
Nov 03 21:26:24.622140 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:24.622112    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3759306a-f917-4d2d-98ee-0bdc38a55460-maas-api-openshift-ca" pod "3759306a-f917-4d2d-98ee-0bdc38a55460" (UID: "3759306a-f917-4d2d-98ee-0bdc38a55460"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:24.623784 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:24.623757    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3759306a-f917-4d2d-98ee-0bdc38a55460-kube-api-access-8dhct" pod "3759306a-f917-4d2d-98ee-0bdc38a55460" (UID: "3759306a-f917-4d2d-98ee-0bdc38a55460"). InnerVolumeSpecName "kube-api-access-8dhct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:24.623878 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:24.623832    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3759306a-f917-4d2d-98ee-0bdc38a55460-maas-api-xks-ca" pod "3759306a-f917-4d2d-98ee-0bdc38a55460" (UID: "3759306a-f917-4d2d-98ee-0bdc38a55460"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:26:24.723154 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:24.723118    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dhct\" (UniqueName: \"kubernetes.io/projected/3759306a-f917-4d2d-98ee-0bdc38a55460-kube-api-access-8dhct\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:24.723154 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:24.723145    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3759306a-f917-4d2d-98ee-0bdc38a55460-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:24.723154 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:24.723154    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3759306a-f917-4d2d-98ee-0bdc38a55460-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:25.395601 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:25.395569    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58"
Nov 03 21:26:25.396081 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:25.395612    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58" event={"ID":"3759306a-f917-4d2d-98ee-0bdc38a55460","Type":"ContainerDied","Data":"6504bf6a2c08d75d37baee5267380269c7e626af7a1bd41d35e121821e524ebc"}
Nov 03 21:26:25.396081 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:25.395682    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6504bf6a2c08d75d37baee5267380269c7e626af7a1bd41d35e121821e524ebc"
Nov 03 21:26:26.375152 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.375118    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz"]
Nov 03 21:26:26.375697 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.375683    5053 cpu_manager.go:414] "containerMap: removing container" podUID="96ddf660-d21e-444b-8567-8de4b71d1227" containerName="revoke-keys"
Nov 03 21:26:26.375751 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.375702    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96ddf660-d21e-444b-8567-8de4b71d1227" containerName="revoke-keys"
Nov 03 21:26:26.375751 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.375714    5053 cpu_manager.go:414] "containerMap: removing container" podUID="3759306a-f917-4d2d-98ee-0bdc38a55460" containerName="revoke-keys"
Nov 03 21:26:26.375751 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.375719    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3759306a-f917-4d2d-98ee-0bdc38a55460" containerName="revoke-keys"
Nov 03 21:26:26.375838 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.375821    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="96ddf660-d21e-444b-8567-8de4b71d1227" containerName="revoke-keys"
Nov 03 21:26:26.375838 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.375830    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="3759306a-f917-4d2d-98ee-0bdc38a55460" containerName="revoke-keys"
Nov 03 21:26:26.378101 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.378086    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz"
Nov 03 21:26:26.434945 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.434913    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz"]
Nov 03 21:26:26.440828 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.440796    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw8kd\" (UniqueName: \"kubernetes.io/projected/89bd6776-d31d-462e-9d1f-3064f55a1c8e-kube-api-access-lw8kd\") pod \"maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz\" (UID: \"89bd6776-d31d-462e-9d1f-3064f55a1c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz"
Nov 03 21:26:26.440947 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.440843    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89bd6776-d31d-462e-9d1f-3064f55a1c8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz\" (UID: \"89bd6776-d31d-462e-9d1f-3064f55a1c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz"
Nov 03 21:26:26.441066 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.441045    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89bd6776-d31d-462e-9d1f-3064f55a1c8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz\" (UID: \"89bd6776-d31d-462e-9d1f-3064f55a1c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz"
Nov 03 21:26:26.480541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.480508    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:26:26.480703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.480546    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:26:26.480703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.480546    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:26:26.480703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.480585    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:26:26.542500 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.542468    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89bd6776-d31d-462e-9d1f-3064f55a1c8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz\" (UID: \"89bd6776-d31d-462e-9d1f-3064f55a1c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz"
Nov 03 21:26:26.542708 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.542557    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw8kd\" (UniqueName: \"kubernetes.io/projected/89bd6776-d31d-462e-9d1f-3064f55a1c8e-kube-api-access-lw8kd\") pod \"maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz\" (UID: \"89bd6776-d31d-462e-9d1f-3064f55a1c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz"
Nov 03 21:26:26.542708 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.542579    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89bd6776-d31d-462e-9d1f-3064f55a1c8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz\" (UID: \"89bd6776-d31d-462e-9d1f-3064f55a1c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz"
Nov 03 21:26:26.543297 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.543269    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89bd6776-d31d-462e-9d1f-3064f55a1c8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz\" (UID: \"89bd6776-d31d-462e-9d1f-3064f55a1c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz"
Nov 03 21:26:26.545091 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.545069    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89bd6776-d31d-462e-9d1f-3064f55a1c8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz\" (UID: \"89bd6776-d31d-462e-9d1f-3064f55a1c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz"
Nov 03 21:26:26.598968 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.598941    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw8kd\" (UniqueName: \"kubernetes.io/projected/89bd6776-d31d-462e-9d1f-3064f55a1c8e-kube-api-access-lw8kd\") pod \"maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz\" (UID: \"89bd6776-d31d-462e-9d1f-3064f55a1c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz"
Nov 03 21:26:26.690150 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.690074    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz"
Nov 03 21:26:26.752078 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.752048    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs"]
Nov 03 21:26:26.757474 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.757452    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs"
Nov 03 21:26:26.809342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.809307    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs"]
Nov 03 21:26:26.845190 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.845148    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa768a1-1de5-4124-a635-605562ef0ab4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs\" (UID: \"7fa768a1-1de5-4124-a635-605562ef0ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs"
Nov 03 21:26:26.845480 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.845211    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa768a1-1de5-4124-a635-605562ef0ab4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs\" (UID: \"7fa768a1-1de5-4124-a635-605562ef0ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs"
Nov 03 21:26:26.845480 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.845364    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8slp5\" (UniqueName: \"kubernetes.io/projected/7fa768a1-1de5-4124-a635-605562ef0ab4-kube-api-access-8slp5\") pod \"maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs\" (UID: \"7fa768a1-1de5-4124-a635-605562ef0ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs"
Nov 03 21:26:26.946331 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.946295    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa768a1-1de5-4124-a635-605562ef0ab4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs\" (UID: \"7fa768a1-1de5-4124-a635-605562ef0ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs"
Nov 03 21:26:26.946584 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.946384    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa768a1-1de5-4124-a635-605562ef0ab4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs\" (UID: \"7fa768a1-1de5-4124-a635-605562ef0ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs"
Nov 03 21:26:26.946584 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.946433    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8slp5\" (UniqueName: \"kubernetes.io/projected/7fa768a1-1de5-4124-a635-605562ef0ab4-kube-api-access-8slp5\") pod \"maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs\" (UID: \"7fa768a1-1de5-4124-a635-605562ef0ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs"
Nov 03 21:26:26.946971 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.946947    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa768a1-1de5-4124-a635-605562ef0ab4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs\" (UID: \"7fa768a1-1de5-4124-a635-605562ef0ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs"
Nov 03 21:26:26.948885 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:26.948860    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa768a1-1de5-4124-a635-605562ef0ab4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs\" (UID: \"7fa768a1-1de5-4124-a635-605562ef0ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs"
Nov 03 21:26:27.002762 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:27.002721    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8slp5\" (UniqueName: \"kubernetes.io/projected/7fa768a1-1de5-4124-a635-605562ef0ab4-kube-api-access-8slp5\") pod \"maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs\" (UID: \"7fa768a1-1de5-4124-a635-605562ef0ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs"
Nov 03 21:26:27.062778 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:27.062742    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz"]
Nov 03 21:26:27.063768 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:26:27.063742    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89bd6776_d31d_462e_9d1f_3064f55a1c8e.slice/crio-b2f0e9bd74f84a3da7849c19ad9798286fa92d71eb5ad5e8442eb25b663f4868 WatchSource:0}: Error finding container b2f0e9bd74f84a3da7849c19ad9798286fa92d71eb5ad5e8442eb25b663f4868: Status 404 returned error can't find the container with id b2f0e9bd74f84a3da7849c19ad9798286fa92d71eb5ad5e8442eb25b663f4868
Nov 03 21:26:27.070822 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:27.070800    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs"
Nov 03 21:26:27.410486 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:27.410455    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz" event={"ID":"89bd6776-d31d-462e-9d1f-3064f55a1c8e","Type":"ContainerStarted","Data":"129ca72db469e4742b67f9f351847a61c21e7ab4dd3483a6060b0cbded223864"}
Nov 03 21:26:27.410652 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:27.410514    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz" event={"ID":"89bd6776-d31d-462e-9d1f-3064f55a1c8e","Type":"ContainerStarted","Data":"b2f0e9bd74f84a3da7849c19ad9798286fa92d71eb5ad5e8442eb25b663f4868"}
Nov 03 21:26:27.464192 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:27.464165    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs"]
Nov 03 21:26:27.465180 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:26:27.465127    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa768a1_1de5_4124_a635_605562ef0ab4.slice/crio-ec8c7edbca560565716580d687c9dad8e50f4217ee2c34d616963084f476988d WatchSource:0}: Error finding container ec8c7edbca560565716580d687c9dad8e50f4217ee2c34d616963084f476988d: Status 404 returned error can't find the container with id ec8c7edbca560565716580d687c9dad8e50f4217ee2c34d616963084f476988d
Nov 03 21:26:27.522842 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:27.522790    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz" podStartSLOduration=1.5227758439999999 podStartE2EDuration="1.522775844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:27.52119606 +0000 UTC m=+702.015117934" watchObservedRunningTime="2026-11-03 21:26:27.522775844 +0000 UTC m=+702.016697719"
Nov 03 21:26:28.297098 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:28.297065    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x"]
Nov 03 21:26:28.417915 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:28.417881    5053 generic.go:356] "Generic (PLEG): container finished" podID="89bd6776-d31d-462e-9d1f-3064f55a1c8e" containerID="129ca72db469e4742b67f9f351847a61c21e7ab4dd3483a6060b0cbded223864" exitCode=0
Nov 03 21:26:28.418108 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:28.417919    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz" event={"ID":"89bd6776-d31d-462e-9d1f-3064f55a1c8e","Type":"ContainerDied","Data":"129ca72db469e4742b67f9f351847a61c21e7ab4dd3483a6060b0cbded223864"}
Nov 03 21:26:28.419553 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:28.419529    5053 generic.go:356] "Generic (PLEG): container finished" podID="7fa768a1-1de5-4124-a635-605562ef0ab4" containerID="631e8bd80dedcb0543220a108644ee40a07bce2dfe0ed35035afc41c38c27f0a" exitCode=0
Nov 03 21:26:28.419677 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:28.419567    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs" event={"ID":"7fa768a1-1de5-4124-a635-605562ef0ab4","Type":"ContainerDied","Data":"631e8bd80dedcb0543220a108644ee40a07bce2dfe0ed35035afc41c38c27f0a"}
Nov 03 21:26:28.419677 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:28.419602    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs" event={"ID":"7fa768a1-1de5-4124-a635-605562ef0ab4","Type":"ContainerStarted","Data":"ec8c7edbca560565716580d687c9dad8e50f4217ee2c34d616963084f476988d"}
Nov 03 21:26:29.576519 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:29.576492    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs"
Nov 03 21:26:29.639999 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:29.639966    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz"
Nov 03 21:26:29.673477 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:29.673454    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fa768a1-1de5-4124-a635-605562ef0ab4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa768a1-1de5-4124-a635-605562ef0ab4-maas-api-xks-ca\") pod \"7fa768a1-1de5-4124-a635-605562ef0ab4\" (UID: \"7fa768a1-1de5-4124-a635-605562ef0ab4\") "
Nov 03 21:26:29.673647 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:29.673528    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa768a1-1de5-4124-a635-605562ef0ab4-kube-api-access-8slp5\" (UniqueName: \"kubernetes.io/projected/7fa768a1-1de5-4124-a635-605562ef0ab4-kube-api-access-8slp5\") pod \"7fa768a1-1de5-4124-a635-605562ef0ab4\" (UID: \"7fa768a1-1de5-4124-a635-605562ef0ab4\") "
Nov 03 21:26:29.673647 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:29.673566    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fa768a1-1de5-4124-a635-605562ef0ab4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa768a1-1de5-4124-a635-605562ef0ab4-maas-api-openshift-ca\") pod \"7fa768a1-1de5-4124-a635-605562ef0ab4\" (UID: \"7fa768a1-1de5-4124-a635-605562ef0ab4\") "
Nov 03 21:26:29.673972 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:29.673947    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa768a1-1de5-4124-a635-605562ef0ab4-maas-api-openshift-ca" pod "7fa768a1-1de5-4124-a635-605562ef0ab4" (UID: "7fa768a1-1de5-4124-a635-605562ef0ab4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:29.675569 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:29.675548    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa768a1-1de5-4124-a635-605562ef0ab4-maas-api-xks-ca" pod "7fa768a1-1de5-4124-a635-605562ef0ab4" (UID: "7fa768a1-1de5-4124-a635-605562ef0ab4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:26:29.675701 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:29.675657    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa768a1-1de5-4124-a635-605562ef0ab4-kube-api-access-8slp5" pod "7fa768a1-1de5-4124-a635-605562ef0ab4" (UID: "7fa768a1-1de5-4124-a635-605562ef0ab4"). InnerVolumeSpecName "kube-api-access-8slp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:29.774590 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:29.774557    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89bd6776-d31d-462e-9d1f-3064f55a1c8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89bd6776-d31d-462e-9d1f-3064f55a1c8e-maas-api-openshift-ca\") pod \"89bd6776-d31d-462e-9d1f-3064f55a1c8e\" (UID: \"89bd6776-d31d-462e-9d1f-3064f55a1c8e\") "
Nov 03 21:26:29.774757 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:29.774699    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89bd6776-d31d-462e-9d1f-3064f55a1c8e-kube-api-access-lw8kd\" (UniqueName: \"kubernetes.io/projected/89bd6776-d31d-462e-9d1f-3064f55a1c8e-kube-api-access-lw8kd\") pod \"89bd6776-d31d-462e-9d1f-3064f55a1c8e\" (UID: \"89bd6776-d31d-462e-9d1f-3064f55a1c8e\") "
Nov 03 21:26:29.774757 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:29.774750    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89bd6776-d31d-462e-9d1f-3064f55a1c8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89bd6776-d31d-462e-9d1f-3064f55a1c8e-maas-api-xks-ca\") pod \"89bd6776-d31d-462e-9d1f-3064f55a1c8e\" (UID: \"89bd6776-d31d-462e-9d1f-3064f55a1c8e\") "
Nov 03 21:26:29.774886 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:29.774864    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89bd6776-d31d-462e-9d1f-3064f55a1c8e-maas-api-openshift-ca" pod "89bd6776-d31d-462e-9d1f-3064f55a1c8e" (UID: "89bd6776-d31d-462e-9d1f-3064f55a1c8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:29.775054 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:29.775039    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa768a1-1de5-4124-a635-605562ef0ab4-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:29.775116 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:29.775054    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89bd6776-d31d-462e-9d1f-3064f55a1c8e-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:29.775116 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:29.775064    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8slp5\" (UniqueName: \"kubernetes.io/projected/7fa768a1-1de5-4124-a635-605562ef0ab4-kube-api-access-8slp5\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:29.775116 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:29.775072    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa768a1-1de5-4124-a635-605562ef0ab4-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:29.776809 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:29.776780    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89bd6776-d31d-462e-9d1f-3064f55a1c8e-kube-api-access-lw8kd" pod "89bd6776-d31d-462e-9d1f-3064f55a1c8e" (UID: "89bd6776-d31d-462e-9d1f-3064f55a1c8e"). InnerVolumeSpecName "kube-api-access-lw8kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:29.776914 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:29.776833    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89bd6776-d31d-462e-9d1f-3064f55a1c8e-maas-api-xks-ca" pod "89bd6776-d31d-462e-9d1f-3064f55a1c8e" (UID: "89bd6776-d31d-462e-9d1f-3064f55a1c8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:26:29.876549 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:29.876520    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw8kd\" (UniqueName: \"kubernetes.io/projected/89bd6776-d31d-462e-9d1f-3064f55a1c8e-kube-api-access-lw8kd\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:29.876549 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:29.876546    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89bd6776-d31d-462e-9d1f-3064f55a1c8e-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:30.432398 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:30.432366    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs"
Nov 03 21:26:30.432398 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:30.432388    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs" event={"ID":"7fa768a1-1de5-4124-a635-605562ef0ab4","Type":"ContainerDied","Data":"ec8c7edbca560565716580d687c9dad8e50f4217ee2c34d616963084f476988d"}
Nov 03 21:26:30.432653 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:30.432440    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec8c7edbca560565716580d687c9dad8e50f4217ee2c34d616963084f476988d"
Nov 03 21:26:30.433730 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:30.433705    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz" event={"ID":"89bd6776-d31d-462e-9d1f-3064f55a1c8e","Type":"ContainerDied","Data":"b2f0e9bd74f84a3da7849c19ad9798286fa92d71eb5ad5e8442eb25b663f4868"}
Nov 03 21:26:30.433858 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:30.433730    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz"
Nov 03 21:26:30.433858 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:30.433748    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2f0e9bd74f84a3da7849c19ad9798286fa92d71eb5ad5e8442eb25b663f4868"
Nov 03 21:26:30.732660 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:30.732627    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr"]
Nov 03 21:26:30.842387 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:30.842354    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz"]
Nov 03 21:26:33.297481 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:33.297436    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x" podUID="30c4b9df-d9fe-4df9-9c03-94049a3e79ce" containerName="payload-pre-processing" containerID="cri-o://7f178403ffcf1491e19c50a3db32854f2976cb00c0a83dbf51f27c83cb665ba6" gracePeriod=25
Nov 03 21:26:33.453927 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:33.453896    5053 generic.go:356] "Generic (PLEG): container finished" podID="30c4b9df-d9fe-4df9-9c03-94049a3e79ce" containerID="7f178403ffcf1491e19c50a3db32854f2976cb00c0a83dbf51f27c83cb665ba6" exitCode=0
Nov 03 21:26:33.454066 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:33.453973    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x" event={"ID":"30c4b9df-d9fe-4df9-9c03-94049a3e79ce","Type":"ContainerDied","Data":"7f178403ffcf1491e19c50a3db32854f2976cb00c0a83dbf51f27c83cb665ba6"}
Nov 03 21:26:33.569505 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:33.569476    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x"
Nov 03 21:26:33.727400 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:33.727370    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c4b9df-d9fe-4df9-9c03-94049a3e79ce-kube-api-access-84sb9\" (UniqueName: \"kubernetes.io/projected/30c4b9df-d9fe-4df9-9c03-94049a3e79ce-kube-api-access-84sb9\") pod \"30c4b9df-d9fe-4df9-9c03-94049a3e79ce\" (UID: \"30c4b9df-d9fe-4df9-9c03-94049a3e79ce\") "
Nov 03 21:26:33.727607 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:33.727512    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30c4b9df-d9fe-4df9-9c03-94049a3e79ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c4b9df-d9fe-4df9-9c03-94049a3e79ce-plugins-config-volume\") pod \"30c4b9df-d9fe-4df9-9c03-94049a3e79ce\" (UID: \"30c4b9df-d9fe-4df9-9c03-94049a3e79ce\") "
Nov 03 21:26:33.727842 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:33.727816    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c4b9df-d9fe-4df9-9c03-94049a3e79ce-plugins-config-volume" pod "30c4b9df-d9fe-4df9-9c03-94049a3e79ce" (UID: "30c4b9df-d9fe-4df9-9c03-94049a3e79ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:33.729668 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:33.729646    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c4b9df-d9fe-4df9-9c03-94049a3e79ce-kube-api-access-84sb9" pod "30c4b9df-d9fe-4df9-9c03-94049a3e79ce" (UID: "30c4b9df-d9fe-4df9-9c03-94049a3e79ce"). InnerVolumeSpecName "kube-api-access-84sb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:33.829382 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:33.829298    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84sb9\" (UniqueName: \"kubernetes.io/projected/30c4b9df-d9fe-4df9-9c03-94049a3e79ce-kube-api-access-84sb9\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:33.829382 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:33.829325    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c4b9df-d9fe-4df9-9c03-94049a3e79ce-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:34.460802 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:34.460772    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x"
Nov 03 21:26:34.461340 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:34.460776    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x" event={"ID":"30c4b9df-d9fe-4df9-9c03-94049a3e79ce","Type":"ContainerDied","Data":"aefcf41cea8758d5a1f0779eb14818ac7acd55ea43f6d6f53f0286723d0a623a"}
Nov 03 21:26:34.461340 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:34.460894    5053 scope.go:122] "RemoveContainer" containerID="7f178403ffcf1491e19c50a3db32854f2976cb00c0a83dbf51f27c83cb665ba6"
Nov 03 21:26:34.624918 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:34.624882    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x"]
Nov 03 21:26:34.628654 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:34.628631    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-889495a7-d7bcb77d7-v267x"]
Nov 03 21:26:35.666611 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:35.666569    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30c4b9df-d9fe-4df9-9c03-94049a3e79ce" path="/var/lib/kubelet/pods/30c4b9df-d9fe-4df9-9c03-94049a3e79ce/volumes"
Nov 03 21:26:35.733868 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:35.733830    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr" podUID="dfa41bfe-320e-4171-abcc-db532e439c78" containerName="payload-processing" containerID="cri-o://6f2c56732d10264a5f0d88cd73d26930a89c1b7809f022fa0da8e823676fb983" gracePeriod=25
Nov 03 21:26:35.842902 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:35.842802    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz" podUID="7f9340d0-88c5-4a7a-a13d-2e28dd451ff8" containerName="payload-pre-processing" containerID="cri-o://32fc6d7632d84e2e733ee25b6069b1b61246c0d14bfa249091cd1b1a0e39f85a" gracePeriod=25
Nov 03 21:26:36.025861 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.025826    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr"
Nov 03 21:26:36.078419 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.078391    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfa41bfe-320e-4171-abcc-db532e439c78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa41bfe-320e-4171-abcc-db532e439c78-plugins-config-volume\") pod \"dfa41bfe-320e-4171-abcc-db532e439c78\" (UID: \"dfa41bfe-320e-4171-abcc-db532e439c78\") "
Nov 03 21:26:36.078576 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.078506    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfa41bfe-320e-4171-abcc-db532e439c78-kube-api-access-887c2\" (UniqueName: \"kubernetes.io/projected/dfa41bfe-320e-4171-abcc-db532e439c78-kube-api-access-887c2\") pod \"dfa41bfe-320e-4171-abcc-db532e439c78\" (UID: \"dfa41bfe-320e-4171-abcc-db532e439c78\") "
Nov 03 21:26:36.078819 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.078794    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa41bfe-320e-4171-abcc-db532e439c78-plugins-config-volume" pod "dfa41bfe-320e-4171-abcc-db532e439c78" (UID: "dfa41bfe-320e-4171-abcc-db532e439c78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:36.081645 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.081619    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa41bfe-320e-4171-abcc-db532e439c78-kube-api-access-887c2" pod "dfa41bfe-320e-4171-abcc-db532e439c78" (UID: "dfa41bfe-320e-4171-abcc-db532e439c78"). InnerVolumeSpecName "kube-api-access-887c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:36.111056 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.111033    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz"
Nov 03 21:26:36.179969 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.179933    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f9340d0-88c5-4a7a-a13d-2e28dd451ff8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f9340d0-88c5-4a7a-a13d-2e28dd451ff8-plugins-config-volume\") pod \"7f9340d0-88c5-4a7a-a13d-2e28dd451ff8\" (UID: \"7f9340d0-88c5-4a7a-a13d-2e28dd451ff8\") "
Nov 03 21:26:36.180134 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.179996    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f9340d0-88c5-4a7a-a13d-2e28dd451ff8-kube-api-access-88pl2\" (UniqueName: \"kubernetes.io/projected/7f9340d0-88c5-4a7a-a13d-2e28dd451ff8-kube-api-access-88pl2\") pod \"7f9340d0-88c5-4a7a-a13d-2e28dd451ff8\" (UID: \"7f9340d0-88c5-4a7a-a13d-2e28dd451ff8\") "
Nov 03 21:26:36.180424 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.180396    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f9340d0-88c5-4a7a-a13d-2e28dd451ff8-plugins-config-volume" pod "7f9340d0-88c5-4a7a-a13d-2e28dd451ff8" (UID: "7f9340d0-88c5-4a7a-a13d-2e28dd451ff8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:36.180543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.180479    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-887c2\" (UniqueName: \"kubernetes.io/projected/dfa41bfe-320e-4171-abcc-db532e439c78-kube-api-access-887c2\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:36.180543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.180500    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa41bfe-320e-4171-abcc-db532e439c78-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:36.182208 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.182184    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f9340d0-88c5-4a7a-a13d-2e28dd451ff8-kube-api-access-88pl2" pod "7f9340d0-88c5-4a7a-a13d-2e28dd451ff8" (UID: "7f9340d0-88c5-4a7a-a13d-2e28dd451ff8"). InnerVolumeSpecName "kube-api-access-88pl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:36.287636 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.282486    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f9340d0-88c5-4a7a-a13d-2e28dd451ff8-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:36.287636 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.282529    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88pl2\" (UniqueName: \"kubernetes.io/projected/7f9340d0-88c5-4a7a-a13d-2e28dd451ff8-kube-api-access-88pl2\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:36.475330 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.475299    5053 generic.go:356] "Generic (PLEG): container finished" podID="7f9340d0-88c5-4a7a-a13d-2e28dd451ff8" containerID="32fc6d7632d84e2e733ee25b6069b1b61246c0d14bfa249091cd1b1a0e39f85a" exitCode=0
Nov 03 21:26:36.475528 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.475343    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz"
Nov 03 21:26:36.475528 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.475387    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz" event={"ID":"7f9340d0-88c5-4a7a-a13d-2e28dd451ff8","Type":"ContainerDied","Data":"32fc6d7632d84e2e733ee25b6069b1b61246c0d14bfa249091cd1b1a0e39f85a"}
Nov 03 21:26:36.475528 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.475438    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz" event={"ID":"7f9340d0-88c5-4a7a-a13d-2e28dd451ff8","Type":"ContainerDied","Data":"daaf9ed5fe044fcda84bd64d4f294be632ccbe6352d479c0c5cd443918ff0336"}
Nov 03 21:26:36.475528 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.475467    5053 scope.go:122] "RemoveContainer" containerID="32fc6d7632d84e2e733ee25b6069b1b61246c0d14bfa249091cd1b1a0e39f85a"
Nov 03 21:26:36.476844 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.476818    5053 generic.go:356] "Generic (PLEG): container finished" podID="dfa41bfe-320e-4171-abcc-db532e439c78" containerID="6f2c56732d10264a5f0d88cd73d26930a89c1b7809f022fa0da8e823676fb983" exitCode=0
Nov 03 21:26:36.477118 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.476858    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr"
Nov 03 21:26:36.477118 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.476906    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr" event={"ID":"dfa41bfe-320e-4171-abcc-db532e439c78","Type":"ContainerDied","Data":"6f2c56732d10264a5f0d88cd73d26930a89c1b7809f022fa0da8e823676fb983"}
Nov 03 21:26:36.477529 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.476956    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr" event={"ID":"dfa41bfe-320e-4171-abcc-db532e439c78","Type":"ContainerDied","Data":"16359ef5e2e90465c96aa07a6f462724f387513c4db5b425761367131a5c9269"}
Nov 03 21:26:36.490498 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.490364    5053 scope.go:122] "RemoveContainer" containerID="32fc6d7632d84e2e733ee25b6069b1b61246c0d14bfa249091cd1b1a0e39f85a"
Nov 03 21:26:36.490838 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:26:36.490811    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32fc6d7632d84e2e733ee25b6069b1b61246c0d14bfa249091cd1b1a0e39f85a\": container with ID starting with 32fc6d7632d84e2e733ee25b6069b1b61246c0d14bfa249091cd1b1a0e39f85a not found: ID does not exist" containerID="32fc6d7632d84e2e733ee25b6069b1b61246c0d14bfa249091cd1b1a0e39f85a"
Nov 03 21:26:36.490944 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.490850    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32fc6d7632d84e2e733ee25b6069b1b61246c0d14bfa249091cd1b1a0e39f85a"} err="failed to get container status \"32fc6d7632d84e2e733ee25b6069b1b61246c0d14bfa249091cd1b1a0e39f85a\": rpc error: code = NotFound desc = could not find container \"32fc6d7632d84e2e733ee25b6069b1b61246c0d14bfa249091cd1b1a0e39f85a\": container with ID starting with 32fc6d7632d84e2e733ee25b6069b1b61246c0d14bfa249091cd1b1a0e39f85a not found: ID does not exist"
Nov 03 21:26:36.491055 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.490994    5053 scope.go:122] "RemoveContainer" containerID="6f2c56732d10264a5f0d88cd73d26930a89c1b7809f022fa0da8e823676fb983"
Nov 03 21:26:36.502393 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.502369    5053 scope.go:122] "RemoveContainer" containerID="6f2c56732d10264a5f0d88cd73d26930a89c1b7809f022fa0da8e823676fb983"
Nov 03 21:26:36.502759 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:26:36.502738    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f2c56732d10264a5f0d88cd73d26930a89c1b7809f022fa0da8e823676fb983\": container with ID starting with 6f2c56732d10264a5f0d88cd73d26930a89c1b7809f022fa0da8e823676fb983 not found: ID does not exist" containerID="6f2c56732d10264a5f0d88cd73d26930a89c1b7809f022fa0da8e823676fb983"
Nov 03 21:26:36.502851 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.502767    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2c56732d10264a5f0d88cd73d26930a89c1b7809f022fa0da8e823676fb983"} err="failed to get container status \"6f2c56732d10264a5f0d88cd73d26930a89c1b7809f022fa0da8e823676fb983\": rpc error: code = NotFound desc = could not find container \"6f2c56732d10264a5f0d88cd73d26930a89c1b7809f022fa0da8e823676fb983\": container with ID starting with 6f2c56732d10264a5f0d88cd73d26930a89c1b7809f022fa0da8e823676fb983 not found: ID does not exist"
Nov 03 21:26:36.637684 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.637600    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr"]
Nov 03 21:26:36.686930 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.686894    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c2e645a4-68b459d6b6-d6nmr"]
Nov 03 21:26:36.741847 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.741800    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-889495a7-bcpl4"]
Nov 03 21:26:36.744162 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.744135    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-889495a7-bcpl4"]
Nov 03 21:26:36.748077 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.748055    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz"]
Nov 03 21:26:36.751885 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:36.751865    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c2e645a4-857498b899-ffltz"]
Nov 03 21:26:37.666045 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:37.666014    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f9340d0-88c5-4a7a-a13d-2e28dd451ff8" path="/var/lib/kubelet/pods/7f9340d0-88c5-4a7a-a13d-2e28dd451ff8/volumes"
Nov 03 21:26:37.666452 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:37.666438    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96ddf660-d21e-444b-8567-8de4b71d1227" path="/var/lib/kubelet/pods/96ddf660-d21e-444b-8567-8de4b71d1227/volumes"
Nov 03 21:26:37.666768 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:37.666757    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfa41bfe-320e-4171-abcc-db532e439c78" path="/var/lib/kubelet/pods/dfa41bfe-320e-4171-abcc-db532e439c78/volumes"
Nov 03 21:26:38.123304 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.123269    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58"]
Nov 03 21:26:38.173041 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.173009    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2e645a4-5mx58"]
Nov 03 21:26:38.223957 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.223924    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"]
Nov 03 21:26:38.224457 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.224442    5053 cpu_manager.go:414] "containerMap: removing container" podUID="dfa41bfe-320e-4171-abcc-db532e439c78" containerName="payload-processing"
Nov 03 21:26:38.224457 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.224458    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfa41bfe-320e-4171-abcc-db532e439c78" containerName="payload-processing"
Nov 03 21:26:38.224544 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.224474    5053 cpu_manager.go:414] "containerMap: removing container" podUID="30c4b9df-d9fe-4df9-9c03-94049a3e79ce" containerName="payload-pre-processing"
Nov 03 21:26:38.224544 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.224479    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c4b9df-d9fe-4df9-9c03-94049a3e79ce" containerName="payload-pre-processing"
Nov 03 21:26:38.224544 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.224494    5053 cpu_manager.go:414] "containerMap: removing container" podUID="7f9340d0-88c5-4a7a-a13d-2e28dd451ff8" containerName="payload-pre-processing"
Nov 03 21:26:38.224544 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.224499    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f9340d0-88c5-4a7a-a13d-2e28dd451ff8" containerName="payload-pre-processing"
Nov 03 21:26:38.224544 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.224510    5053 cpu_manager.go:414] "containerMap: removing container" podUID="7fa768a1-1de5-4124-a635-605562ef0ab4" containerName="revoke-keys"
Nov 03 21:26:38.224544 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.224514    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fa768a1-1de5-4124-a635-605562ef0ab4" containerName="revoke-keys"
Nov 03 21:26:38.224544 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.224521    5053 cpu_manager.go:414] "containerMap: removing container" podUID="89bd6776-d31d-462e-9d1f-3064f55a1c8e" containerName="revoke-keys"
Nov 03 21:26:38.224544 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.224525    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89bd6776-d31d-462e-9d1f-3064f55a1c8e" containerName="revoke-keys"
Nov 03 21:26:38.224890 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.224609    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fa768a1-1de5-4124-a635-605562ef0ab4" containerName="revoke-keys"
Nov 03 21:26:38.224890 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.224617    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="89bd6776-d31d-462e-9d1f-3064f55a1c8e" containerName="revoke-keys"
Nov 03 21:26:38.224890 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.224627    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c4b9df-d9fe-4df9-9c03-94049a3e79ce" containerName="payload-pre-processing"
Nov 03 21:26:38.224890 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.224637    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfa41bfe-320e-4171-abcc-db532e439c78" containerName="payload-processing"
Nov 03 21:26:38.224890 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.224650    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f9340d0-88c5-4a7a-a13d-2e28dd451ff8" containerName="payload-pre-processing"
Nov 03 21:26:38.229481 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.229465    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.240806 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.240780    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw"]
Nov 03 21:26:38.299987 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.291432    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-d71a74-openshift-default-dockercfg-nb5n6\""
Nov 03 21:26:38.299987 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.294641    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"]
Nov 03 21:26:38.312939 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.312907    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.313119 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.313028    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.313222 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.313140    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.313222 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.313182    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.313389 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.313298    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.313450 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.313396    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e0381b-b02d-40f1-acef-babc6c0248d1-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.313450 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.313439    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-786pf\" (UniqueName: \"kubernetes.io/projected/88e0381b-b02d-40f1-acef-babc6c0248d1-kube-api-access-786pf\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.313554 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.313531    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.313609 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.313597    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.345641 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.345564    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc"]
Nov 03 21:26:38.414905 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.414864    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.415114 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.414920    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.415114 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.415094    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.415283 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.415145    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.415283 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.415179    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.415283 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.415223    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.415444 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.415336    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e0381b-b02d-40f1-acef-babc6c0248d1-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.415444 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.415364    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-786pf\" (UniqueName: \"kubernetes.io/projected/88e0381b-b02d-40f1-acef-babc6c0248d1-kube-api-access-786pf\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.415444 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.415407    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.415444 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.415426    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.415624 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.415557    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.415682 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.415641    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.415682 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.415655    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.416218 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.416196    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e0381b-b02d-40f1-acef-babc6c0248d1-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.418012 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.417809    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.418147 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.418024    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.480998 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.480944    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.482210 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.482175    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-786pf\" (UniqueName: \"kubernetes.io/projected/88e0381b-b02d-40f1-acef-babc6c0248d1-kube-api-access-786pf\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-868tj\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.542076 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.542040    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:38.668896 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.668863    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"]
Nov 03 21:26:38.675946 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.675903    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd" podUID="a253cf6d-2f11-4dcb-b788-116090cdfe8b" containerName="istio-proxy" containerID="cri-o://a6811d397649cfdb0dcf18b13814896cec308d96b151a517691e0194d8e5e97a" gracePeriod=30
Nov 03 21:26:38.828448 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.828415    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8"]
Nov 03 21:26:38.838959 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.838926    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8"
Nov 03 21:26:38.888069 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.887932    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8"]
Nov 03 21:26:38.890936 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.890907    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:26:38.891099 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.891022    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:26:38.896360 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.896327    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:26:38.896676 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.896654    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:26:38.920294 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.920253    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmgl2\" (UniqueName: \"kubernetes.io/projected/19e966b7-8a93-48ed-85eb-91dc76651258-kube-api-access-xmgl2\") pod \"maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8\" (UID: \"19e966b7-8a93-48ed-85eb-91dc76651258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8"
Nov 03 21:26:38.920474 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.920446    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e966b7-8a93-48ed-85eb-91dc76651258-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8\" (UID: \"19e966b7-8a93-48ed-85eb-91dc76651258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8"
Nov 03 21:26:38.920541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.920499    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e966b7-8a93-48ed-85eb-91dc76651258-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8\" (UID: \"19e966b7-8a93-48ed-85eb-91dc76651258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8"
Nov 03 21:26:38.989943 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.989916    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"]
Nov 03 21:26:38.991423 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:26:38.991397    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e0381b_b02d_40f1_acef_babc6c0248d1.slice/crio-6731326ae61c08506448aa9202b432f4e5571d28ecb72ba72739097c46d96683 WatchSource:0}: Error finding container 6731326ae61c08506448aa9202b432f4e5571d28ecb72ba72739097c46d96683: Status 404 returned error can't find the container with id 6731326ae61c08506448aa9202b432f4e5571d28ecb72ba72739097c46d96683
Nov 03 21:26:38.993586 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.993545    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:26:38.993682 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.993616    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:26:38.993682 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:38.993648    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:26:39.022036 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:39.022008    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e966b7-8a93-48ed-85eb-91dc76651258-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8\" (UID: \"19e966b7-8a93-48ed-85eb-91dc76651258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8"
Nov 03 21:26:39.022184 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:39.022067    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e966b7-8a93-48ed-85eb-91dc76651258-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8\" (UID: \"19e966b7-8a93-48ed-85eb-91dc76651258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8"
Nov 03 21:26:39.022184 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:39.022123    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmgl2\" (UniqueName: \"kubernetes.io/projected/19e966b7-8a93-48ed-85eb-91dc76651258-kube-api-access-xmgl2\") pod \"maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8\" (UID: \"19e966b7-8a93-48ed-85eb-91dc76651258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8"
Nov 03 21:26:39.022897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:39.022816    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e966b7-8a93-48ed-85eb-91dc76651258-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8\" (UID: \"19e966b7-8a93-48ed-85eb-91dc76651258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8"
Nov 03 21:26:39.024455 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:39.024435    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e966b7-8a93-48ed-85eb-91dc76651258-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8\" (UID: \"19e966b7-8a93-48ed-85eb-91dc76651258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8"
Nov 03 21:26:39.080682 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:39.080652    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmgl2\" (UniqueName: \"kubernetes.io/projected/19e966b7-8a93-48ed-85eb-91dc76651258-kube-api-access-xmgl2\") pod \"maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8\" (UID: \"19e966b7-8a93-48ed-85eb-91dc76651258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8"
Nov 03 21:26:39.152078 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:39.151990    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8"
Nov 03 21:26:39.498648 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:39.498611    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj" event={"ID":"88e0381b-b02d-40f1-acef-babc6c0248d1","Type":"ContainerStarted","Data":"6c9b7abbf29d54e93d7391907f9f88d1229fa2e85431be99a15107ad56df6380"}
Nov 03 21:26:39.498849 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:39.498670    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj" event={"ID":"88e0381b-b02d-40f1-acef-babc6c0248d1","Type":"ContainerStarted","Data":"6731326ae61c08506448aa9202b432f4e5571d28ecb72ba72739097c46d96683"}
Nov 03 21:26:39.522207 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:39.522182    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8"]
Nov 03 21:26:39.524597 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:26:39.524568    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e966b7_8a93_48ed_85eb_91dc76651258.slice/crio-f04b1afe42b526a0bd2e601c30a37b072947a45413b5f476b9d96d2b666c0079 WatchSource:0}: Error finding container f04b1afe42b526a0bd2e601c30a37b072947a45413b5f476b9d96d2b666c0079: Status 404 returned error can't find the container with id f04b1afe42b526a0bd2e601c30a37b072947a45413b5f476b9d96d2b666c0079
Nov 03 21:26:39.544753 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:39.542373    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:39.616505 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:39.616445    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj" podStartSLOduration=1.6164267460000001 podStartE2EDuration="1.616426746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:39.615287099 +0000 UTC m=+714.109208976" watchObservedRunningTime="2026-11-03 21:26:39.616426746 +0000 UTC m=+714.110348622"
Nov 03 21:26:39.665951 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:39.665919    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3759306a-f917-4d2d-98ee-0bdc38a55460" path="/var/lib/kubelet/pods/3759306a-f917-4d2d-98ee-0bdc38a55460/volumes"
Nov 03 21:26:40.506856 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:40.506816    5053 generic.go:356] "Generic (PLEG): container finished" podID="19e966b7-8a93-48ed-85eb-91dc76651258" containerID="29ca2d4c063643916c488d76af3018d92e4feeee01d432e6237fa54d6c9e1140" exitCode=0
Nov 03 21:26:40.507293 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:40.506907    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8" event={"ID":"19e966b7-8a93-48ed-85eb-91dc76651258","Type":"ContainerDied","Data":"29ca2d4c063643916c488d76af3018d92e4feeee01d432e6237fa54d6c9e1140"}
Nov 03 21:26:40.507293 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:40.506964    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8" event={"ID":"19e966b7-8a93-48ed-85eb-91dc76651258","Type":"ContainerStarted","Data":"f04b1afe42b526a0bd2e601c30a37b072947a45413b5f476b9d96d2b666c0079"}
Nov 03 21:26:40.548526 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:40.548499    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:40.931036 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:26:40.930948    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:26:41.513504 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:41.513460    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:41.514618 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:41.514597    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:26:41.674729 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:41.674701    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8"
Nov 03 21:26:41.756394 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:41.756360    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19e966b7-8a93-48ed-85eb-91dc76651258-kube-api-access-xmgl2\" (UniqueName: \"kubernetes.io/projected/19e966b7-8a93-48ed-85eb-91dc76651258-kube-api-access-xmgl2\") pod \"19e966b7-8a93-48ed-85eb-91dc76651258\" (UID: \"19e966b7-8a93-48ed-85eb-91dc76651258\") "
Nov 03 21:26:41.756588 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:41.756413    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19e966b7-8a93-48ed-85eb-91dc76651258-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e966b7-8a93-48ed-85eb-91dc76651258-maas-api-openshift-ca\") pod \"19e966b7-8a93-48ed-85eb-91dc76651258\" (UID: \"19e966b7-8a93-48ed-85eb-91dc76651258\") "
Nov 03 21:26:41.756588 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:41.756492    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19e966b7-8a93-48ed-85eb-91dc76651258-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e966b7-8a93-48ed-85eb-91dc76651258-maas-api-xks-ca\") pod \"19e966b7-8a93-48ed-85eb-91dc76651258\" (UID: \"19e966b7-8a93-48ed-85eb-91dc76651258\") "
Nov 03 21:26:41.756817 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:41.756793    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19e966b7-8a93-48ed-85eb-91dc76651258-maas-api-openshift-ca" pod "19e966b7-8a93-48ed-85eb-91dc76651258" (UID: "19e966b7-8a93-48ed-85eb-91dc76651258"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:41.758594 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:41.758574    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19e966b7-8a93-48ed-85eb-91dc76651258-maas-api-xks-ca" pod "19e966b7-8a93-48ed-85eb-91dc76651258" (UID: "19e966b7-8a93-48ed-85eb-91dc76651258"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:26:41.758669 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:41.758643    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e966b7-8a93-48ed-85eb-91dc76651258-kube-api-access-xmgl2" pod "19e966b7-8a93-48ed-85eb-91dc76651258" (UID: "19e966b7-8a93-48ed-85eb-91dc76651258"). InnerVolumeSpecName "kube-api-access-xmgl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:41.858037 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:41.857996    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmgl2\" (UniqueName: \"kubernetes.io/projected/19e966b7-8a93-48ed-85eb-91dc76651258-kube-api-access-xmgl2\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:41.858037 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:41.858029    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e966b7-8a93-48ed-85eb-91dc76651258-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:41.858037 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:41.858044    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e966b7-8a93-48ed-85eb-91dc76651258-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:42.076420 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:42.076334    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7"]
Nov 03 21:26:42.520844 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:42.520818    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8"
Nov 03 21:26:42.521255 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:42.520850    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8" event={"ID":"19e966b7-8a93-48ed-85eb-91dc76651258","Type":"ContainerDied","Data":"f04b1afe42b526a0bd2e601c30a37b072947a45413b5f476b9d96d2b666c0079"}
Nov 03 21:26:42.521255 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:42.520895    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f04b1afe42b526a0bd2e601c30a37b072947a45413b5f476b9d96d2b666c0079"
Nov 03 21:26:43.241628 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.241581    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw" podUID="17334f9f-d4ee-4e20-85d8-f211ed2d6c62" containerName="payload-processing" containerID="cri-o://4d83383e87d16da6ae10c0ff92ebcc63e7ecdbd187657a7176403ec40abe256f" gracePeriod=25
Nov 03 21:26:43.346578 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.346527    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc" podUID="d6ffe32b-429e-4e1a-bdf3-6e30f287f6df" containerName="payload-pre-processing" containerID="cri-o://cd50c2cd445015fa79bc8405518449201ce84b4c49aefd9bebbc436bff3c54cf" gracePeriod=25
Nov 03 21:26:43.528786 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.528751    5053 generic.go:356] "Generic (PLEG): container finished" podID="17334f9f-d4ee-4e20-85d8-f211ed2d6c62" containerID="4d83383e87d16da6ae10c0ff92ebcc63e7ecdbd187657a7176403ec40abe256f" exitCode=0
Nov 03 21:26:43.529125 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.528840    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw" event={"ID":"17334f9f-d4ee-4e20-85d8-f211ed2d6c62","Type":"ContainerDied","Data":"4d83383e87d16da6ae10c0ff92ebcc63e7ecdbd187657a7176403ec40abe256f"}
Nov 03 21:26:43.529125 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.528907    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw" event={"ID":"17334f9f-d4ee-4e20-85d8-f211ed2d6c62","Type":"ContainerDied","Data":"69d4b7dd6dbb863f34fd3486cf7438ef515e41de5266f4b7e8499053690657c1"}
Nov 03 21:26:43.529125 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.528928    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69d4b7dd6dbb863f34fd3486cf7438ef515e41de5266f4b7e8499053690657c1"
Nov 03 21:26:43.530585 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.530560    5053 generic.go:356] "Generic (PLEG): container finished" podID="d6ffe32b-429e-4e1a-bdf3-6e30f287f6df" containerID="cd50c2cd445015fa79bc8405518449201ce84b4c49aefd9bebbc436bff3c54cf" exitCode=0
Nov 03 21:26:43.530698 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.530612    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc" event={"ID":"d6ffe32b-429e-4e1a-bdf3-6e30f287f6df","Type":"ContainerDied","Data":"cd50c2cd445015fa79bc8405518449201ce84b4c49aefd9bebbc436bff3c54cf"}
Nov 03 21:26:43.553264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.553208    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw"
Nov 03 21:26:43.681293 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.667602    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc"
Nov 03 21:26:43.691711 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.691683    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17334f9f-d4ee-4e20-85d8-f211ed2d6c62-kube-api-access-bmqbz\" (UniqueName: \"kubernetes.io/projected/17334f9f-d4ee-4e20-85d8-f211ed2d6c62-kube-api-access-bmqbz\") pod \"17334f9f-d4ee-4e20-85d8-f211ed2d6c62\" (UID: \"17334f9f-d4ee-4e20-85d8-f211ed2d6c62\") "
Nov 03 21:26:43.691869 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.691734    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17334f9f-d4ee-4e20-85d8-f211ed2d6c62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17334f9f-d4ee-4e20-85d8-f211ed2d6c62-plugins-config-volume\") pod \"17334f9f-d4ee-4e20-85d8-f211ed2d6c62\" (UID: \"17334f9f-d4ee-4e20-85d8-f211ed2d6c62\") "
Nov 03 21:26:43.692332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.692304    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17334f9f-d4ee-4e20-85d8-f211ed2d6c62-plugins-config-volume" pod "17334f9f-d4ee-4e20-85d8-f211ed2d6c62" (UID: "17334f9f-d4ee-4e20-85d8-f211ed2d6c62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:43.694166 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.694139    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17334f9f-d4ee-4e20-85d8-f211ed2d6c62-kube-api-access-bmqbz" pod "17334f9f-d4ee-4e20-85d8-f211ed2d6c62" (UID: "17334f9f-d4ee-4e20-85d8-f211ed2d6c62"). InnerVolumeSpecName "kube-api-access-bmqbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:43.792570 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.792544    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6ffe32b-429e-4e1a-bdf3-6e30f287f6df-kube-api-access-tnvtf\" (UniqueName: \"kubernetes.io/projected/d6ffe32b-429e-4e1a-bdf3-6e30f287f6df-kube-api-access-tnvtf\") pod \"d6ffe32b-429e-4e1a-bdf3-6e30f287f6df\" (UID: \"d6ffe32b-429e-4e1a-bdf3-6e30f287f6df\") "
Nov 03 21:26:43.792695 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.792581    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6ffe32b-429e-4e1a-bdf3-6e30f287f6df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6ffe32b-429e-4e1a-bdf3-6e30f287f6df-plugins-config-volume\") pod \"d6ffe32b-429e-4e1a-bdf3-6e30f287f6df\" (UID: \"d6ffe32b-429e-4e1a-bdf3-6e30f287f6df\") "
Nov 03 21:26:43.792812 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.792797    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmqbz\" (UniqueName: \"kubernetes.io/projected/17334f9f-d4ee-4e20-85d8-f211ed2d6c62-kube-api-access-bmqbz\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:43.792851 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.792815    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17334f9f-d4ee-4e20-85d8-f211ed2d6c62-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:43.793035 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.793004    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6ffe32b-429e-4e1a-bdf3-6e30f287f6df-plugins-config-volume" pod "d6ffe32b-429e-4e1a-bdf3-6e30f287f6df" (UID: "d6ffe32b-429e-4e1a-bdf3-6e30f287f6df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:43.794903 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.794873    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ffe32b-429e-4e1a-bdf3-6e30f287f6df-kube-api-access-tnvtf" pod "d6ffe32b-429e-4e1a-bdf3-6e30f287f6df" (UID: "d6ffe32b-429e-4e1a-bdf3-6e30f287f6df"). InnerVolumeSpecName "kube-api-access-tnvtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:43.893861 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.893830    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnvtf\" (UniqueName: \"kubernetes.io/projected/d6ffe32b-429e-4e1a-bdf3-6e30f287f6df-kube-api-access-tnvtf\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:43.893861 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.893862    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6ffe32b-429e-4e1a-bdf3-6e30f287f6df-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:43.917929 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.917905    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:26:43.994964 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.994925    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-credential-socket\") pod \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") "
Nov 03 21:26:43.995135 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.994983    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-podinfo\") pod \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") "
Nov 03 21:26:43.995135 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.995030    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-data\") pod \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") "
Nov 03 21:26:43.995135 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.995075    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-workload-certs\") pod \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") "
Nov 03 21:26:43.995326 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.995142    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-envoy\") pod \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") "
Nov 03 21:26:43.995326 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.995178    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-workload-socket\") pod \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") "
Nov 03 21:26:43.995326 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.995221    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istiod-ca-cert\") pod \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") "
Nov 03 21:26:43.995326 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.995224    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-credential-socket" pod "a253cf6d-2f11-4dcb-b788-116090cdfe8b" (UID: "a253cf6d-2f11-4dcb-b788-116090cdfe8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:26:43.995326 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.995300    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a253cf6d-2f11-4dcb-b788-116090cdfe8b-kube-api-access-r49xb\" (UniqueName: \"kubernetes.io/projected/a253cf6d-2f11-4dcb-b788-116090cdfe8b-kube-api-access-r49xb\") pod \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") "
Nov 03 21:26:43.995552 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.995334    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-token\" (UniqueName: \"kubernetes.io/projected/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-token\") pod \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\" (UID: \"a253cf6d-2f11-4dcb-b788-116090cdfe8b\") "
Nov 03 21:26:43.995552 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.995375    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-data" pod "a253cf6d-2f11-4dcb-b788-116090cdfe8b" (UID: "a253cf6d-2f11-4dcb-b788-116090cdfe8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:26:43.995552 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.995455    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-workload-certs" pod "a253cf6d-2f11-4dcb-b788-116090cdfe8b" (UID: "a253cf6d-2f11-4dcb-b788-116090cdfe8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:26:43.995707 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.995514    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-workload-socket" pod "a253cf6d-2f11-4dcb-b788-116090cdfe8b" (UID: "a253cf6d-2f11-4dcb-b788-116090cdfe8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:26:43.995707 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.995678    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istiod-ca-cert" pod "a253cf6d-2f11-4dcb-b788-116090cdfe8b" (UID: "a253cf6d-2f11-4dcb-b788-116090cdfe8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:43.995945 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.995844    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:43.996017 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.995982    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:43.996017 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.995997    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:43.996017 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.996010    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:43.996119 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.996022    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:43.997404 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.997375    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-podinfo" pod "a253cf6d-2f11-4dcb-b788-116090cdfe8b" (UID: "a253cf6d-2f11-4dcb-b788-116090cdfe8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:26:43.997404 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.997400    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-envoy" pod "a253cf6d-2f11-4dcb-b788-116090cdfe8b" (UID: "a253cf6d-2f11-4dcb-b788-116090cdfe8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:26:43.997905 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.997887    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a253cf6d-2f11-4dcb-b788-116090cdfe8b-kube-api-access-r49xb" pod "a253cf6d-2f11-4dcb-b788-116090cdfe8b" (UID: "a253cf6d-2f11-4dcb-b788-116090cdfe8b"). InnerVolumeSpecName "kube-api-access-r49xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:43.997943 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:43.997916    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-token" pod "a253cf6d-2f11-4dcb-b788-116090cdfe8b" (UID: "a253cf6d-2f11-4dcb-b788-116090cdfe8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:44.097486 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.097406    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:44.097486 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.097433    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r49xb\" (UniqueName: \"kubernetes.io/projected/a253cf6d-2f11-4dcb-b788-116090cdfe8b-kube-api-access-r49xb\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:44.097486 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.097444    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:44.097486 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.097454    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a253cf6d-2f11-4dcb-b788-116090cdfe8b-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:44.445175 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.445122    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz"]
Nov 03 21:26:44.447746 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.447719    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e7203b81-9zgzz"]
Nov 03 21:26:44.521501 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.521470    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"]
Nov 03 21:26:44.521733 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.521709    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td" podUID="0013fad5-118c-4e7b-b51f-6fdb78d47bb4" containerName="istio-proxy" containerID="cri-o://bc65cfdacbd84cec9a6ca6d545cc1d151c447a0e853692de19931dec6f822430" gracePeriod=30
Nov 03 21:26:44.539055 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.539025    5053 generic.go:356] "Generic (PLEG): container finished" podID="a253cf6d-2f11-4dcb-b788-116090cdfe8b" containerID="a6811d397649cfdb0dcf18b13814896cec308d96b151a517691e0194d8e5e97a" exitCode=0
Nov 03 21:26:44.539647 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.539099    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"
Nov 03 21:26:44.539647 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.539114    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd" event={"ID":"a253cf6d-2f11-4dcb-b788-116090cdfe8b","Type":"ContainerDied","Data":"a6811d397649cfdb0dcf18b13814896cec308d96b151a517691e0194d8e5e97a"}
Nov 03 21:26:44.539647 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.539180    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd" event={"ID":"a253cf6d-2f11-4dcb-b788-116090cdfe8b","Type":"ContainerDied","Data":"f9b25a3a19c6b0f299e44099b255e2d2a056a09c2311896048f272bd3c78ca64"}
Nov 03 21:26:44.539647 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.539210    5053 scope.go:122] "RemoveContainer" containerID="a6811d397649cfdb0dcf18b13814896cec308d96b151a517691e0194d8e5e97a"
Nov 03 21:26:44.541578 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.541557    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw"
Nov 03 21:26:44.543389 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.543365    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc"
Nov 03 21:26:44.547414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.547359    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc" event={"ID":"d6ffe32b-429e-4e1a-bdf3-6e30f287f6df","Type":"ContainerDied","Data":"0a3bd6d4f9c1ed10aa7f942144baf334ee69ca731dcc6b7cd441af0999f19ba4"}
Nov 03 21:26:44.549681 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.549663    5053 scope.go:122] "RemoveContainer" containerID="07f2b805c892b58e80dd721b3defe9a14e67ce3d68ed4d7ed21fc72cb286eace"
Nov 03 21:26:44.559392 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.559378    5053 scope.go:122] "RemoveContainer" containerID="a6811d397649cfdb0dcf18b13814896cec308d96b151a517691e0194d8e5e97a"
Nov 03 21:26:44.559747 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:26:44.559724    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6811d397649cfdb0dcf18b13814896cec308d96b151a517691e0194d8e5e97a\": container with ID starting with a6811d397649cfdb0dcf18b13814896cec308d96b151a517691e0194d8e5e97a not found: ID does not exist" containerID="a6811d397649cfdb0dcf18b13814896cec308d96b151a517691e0194d8e5e97a"
Nov 03 21:26:44.559847 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.559760    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6811d397649cfdb0dcf18b13814896cec308d96b151a517691e0194d8e5e97a"} err="failed to get container status \"a6811d397649cfdb0dcf18b13814896cec308d96b151a517691e0194d8e5e97a\": rpc error: code = NotFound desc = could not find container \"a6811d397649cfdb0dcf18b13814896cec308d96b151a517691e0194d8e5e97a\": container with ID starting with a6811d397649cfdb0dcf18b13814896cec308d96b151a517691e0194d8e5e97a not found: ID does not exist"
Nov 03 21:26:44.559847 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.559784    5053 scope.go:122] "RemoveContainer" containerID="07f2b805c892b58e80dd721b3defe9a14e67ce3d68ed4d7ed21fc72cb286eace"
Nov 03 21:26:44.560113 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:26:44.560090    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07f2b805c892b58e80dd721b3defe9a14e67ce3d68ed4d7ed21fc72cb286eace\": container with ID starting with 07f2b805c892b58e80dd721b3defe9a14e67ce3d68ed4d7ed21fc72cb286eace not found: ID does not exist" containerID="07f2b805c892b58e80dd721b3defe9a14e67ce3d68ed4d7ed21fc72cb286eace"
Nov 03 21:26:44.560218 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.560116    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07f2b805c892b58e80dd721b3defe9a14e67ce3d68ed4d7ed21fc72cb286eace"} err="failed to get container status \"07f2b805c892b58e80dd721b3defe9a14e67ce3d68ed4d7ed21fc72cb286eace\": rpc error: code = NotFound desc = could not find container \"07f2b805c892b58e80dd721b3defe9a14e67ce3d68ed4d7ed21fc72cb286eace\": container with ID starting with 07f2b805c892b58e80dd721b3defe9a14e67ce3d68ed4d7ed21fc72cb286eace not found: ID does not exist"
Nov 03 21:26:44.560218 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.560132    5053 scope.go:122] "RemoveContainer" containerID="cd50c2cd445015fa79bc8405518449201ce84b4c49aefd9bebbc436bff3c54cf"
Nov 03 21:26:44.766372 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.766293    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw"]
Nov 03 21:26:44.770567 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.770524    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e7203b81-7bc6579f68-mw9rw"]
Nov 03 21:26:44.880443 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.880404    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"]
Nov 03 21:26:44.887312 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.887280    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c2e645a4-openshift-default-68b77dc59c-gv8gd"]
Nov 03 21:26:44.999117 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:44.999074    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc"]
Nov 03 21:26:45.001307 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:45.001281    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e7203b81-6554b474c9-4f5lc"]
Nov 03 21:26:45.665619 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:45.665590    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17334f9f-d4ee-4e20-85d8-f211ed2d6c62" path="/var/lib/kubelet/pods/17334f9f-d4ee-4e20-85d8-f211ed2d6c62/volumes"
Nov 03 21:26:45.666008 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:45.665997    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89bd6776-d31d-462e-9d1f-3064f55a1c8e" path="/var/lib/kubelet/pods/89bd6776-d31d-462e-9d1f-3064f55a1c8e/volumes"
Nov 03 21:26:45.666390 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:45.666376    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a253cf6d-2f11-4dcb-b788-116090cdfe8b" path="/var/lib/kubelet/pods/a253cf6d-2f11-4dcb-b788-116090cdfe8b/volumes"
Nov 03 21:26:45.666983 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:45.666968    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6ffe32b-429e-4e1a-bdf3-6e30f287f6df" path="/var/lib/kubelet/pods/d6ffe32b-429e-4e1a-bdf3-6e30f287f6df/volumes"
Nov 03 21:26:45.940829 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:45.940754    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"]
Nov 03 21:26:45.941315 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:45.941296    5053 cpu_manager.go:414] "containerMap: removing container" podUID="d6ffe32b-429e-4e1a-bdf3-6e30f287f6df" containerName="payload-pre-processing"
Nov 03 21:26:45.941315 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:45.941312    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6ffe32b-429e-4e1a-bdf3-6e30f287f6df" containerName="payload-pre-processing"
Nov 03 21:26:45.941494 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:45.941324    5053 cpu_manager.go:414] "containerMap: removing container" podUID="a253cf6d-2f11-4dcb-b788-116090cdfe8b" containerName="istio-proxy"
Nov 03 21:26:45.941494 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:45.941329    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a253cf6d-2f11-4dcb-b788-116090cdfe8b" containerName="istio-proxy"
Nov 03 21:26:45.941494 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:45.941339    5053 cpu_manager.go:414] "containerMap: removing container" podUID="19e966b7-8a93-48ed-85eb-91dc76651258" containerName="revoke-keys"
Nov 03 21:26:45.941494 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:45.941343    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19e966b7-8a93-48ed-85eb-91dc76651258" containerName="revoke-keys"
Nov 03 21:26:45.941494 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:45.941376    5053 cpu_manager.go:414] "containerMap: removing container" podUID="17334f9f-d4ee-4e20-85d8-f211ed2d6c62" containerName="payload-processing"
Nov 03 21:26:45.941494 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:45.941381    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17334f9f-d4ee-4e20-85d8-f211ed2d6c62" containerName="payload-processing"
Nov 03 21:26:45.941770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:45.941500    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="19e966b7-8a93-48ed-85eb-91dc76651258" containerName="revoke-keys"
Nov 03 21:26:45.941770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:45.941518    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="17334f9f-d4ee-4e20-85d8-f211ed2d6c62" containerName="payload-processing"
Nov 03 21:26:45.941770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:45.941531    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="a253cf6d-2f11-4dcb-b788-116090cdfe8b" containerName="istio-proxy"
Nov 03 21:26:45.941770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:45.941541    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6ffe32b-429e-4e1a-bdf3-6e30f287f6df" containerName="payload-pre-processing"
Nov 03 21:26:45.941770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:45.941549    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="a253cf6d-2f11-4dcb-b788-116090cdfe8b" containerName="istio-proxy"
Nov 03 21:26:45.946435 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:45.946416    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.011306 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.009649    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-da5f0b-openshift-default-dockercfg-nvv9f\""
Nov 03 21:26:46.023057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.017780    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-data\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.023057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.017859    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-workload-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.023057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.017919    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-podinfo\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.023057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.017952    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-credential-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.023057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.017983    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-workload-certs\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.023057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.018022    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5znl\" (UniqueName: \"kubernetes.io/projected/0a1fdc06-686e-499e-9c23-ae901773d3b6-kube-api-access-n5znl\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.023057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.018077    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-token\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.023057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.018148    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-envoy\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.023057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.018211    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1fdc06-686e-499e-9c23-ae901773d3b6-istiod-ca-cert\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.061968 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.061936    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"]
Nov 03 21:26:46.119266 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.119127    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5znl\" (UniqueName: \"kubernetes.io/projected/0a1fdc06-686e-499e-9c23-ae901773d3b6-kube-api-access-n5znl\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.119423 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.119378    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-token\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.119477 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.119443    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-envoy\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.119567 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.119485    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1fdc06-686e-499e-9c23-ae901773d3b6-istiod-ca-cert\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.119567 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.119516    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-data\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.119567 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.119548    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-workload-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.119720 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.119580    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-podinfo\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.119720 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.119601    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-credential-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.119720 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.119627    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-workload-certs\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.120005 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.119980    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-workload-certs\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.120107 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.120009    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-data\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.120107 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.120090    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-workload-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.120292 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.120271    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-credential-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.120507 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.120487    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1fdc06-686e-499e-9c23-ae901773d3b6-istiod-ca-cert\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.121973 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.121933    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-envoy\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.122709 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.122682    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-podinfo\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.181345 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.181311    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-token\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.181473 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.181382    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5znl\" (UniqueName: \"kubernetes.io/projected/0a1fdc06-686e-499e-9c23-ae901773d3b6-kube-api-access-n5znl\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.259404 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.259369    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:46.697305 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.697271    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"]
Nov 03 21:26:46.698641 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:26:46.698610    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a1fdc06_686e_499e_9c23_ae901773d3b6.slice/crio-2a81d1e779df173196eeb68fbc09984220e4f3e50a1899a86659144007c04930 WatchSource:0}: Error finding container 2a81d1e779df173196eeb68fbc09984220e4f3e50a1899a86659144007c04930: Status 404 returned error can't find the container with id 2a81d1e779df173196eeb68fbc09984220e4f3e50a1899a86659144007c04930
Nov 03 21:26:46.701203 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.701170    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:26:46.701296 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.701260    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:26:46.701331 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:46.701294    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:26:47.035084 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.035006    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr"]
Nov 03 21:26:47.035565 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.035550    5053 cpu_manager.go:414] "containerMap: removing container" podUID="a253cf6d-2f11-4dcb-b788-116090cdfe8b" containerName="istio-proxy"
Nov 03 21:26:47.035623 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.035567    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a253cf6d-2f11-4dcb-b788-116090cdfe8b" containerName="istio-proxy"
Nov 03 21:26:47.038921 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.038902    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr"
Nov 03 21:26:47.076612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.076566    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7" podUID="42e8cb6e-820e-45d6-8af3-fa981f2bfa7a" containerName="payload-pre-processing" containerID="cri-o://ddebab7a55a532c0ee1a1b4712bbc6ce4445b253d308d3950b84e7e64b2a1a17" gracePeriod=25
Nov 03 21:26:47.092556 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.092508    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-d71a74\""
Nov 03 21:26:47.092682 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.092635    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-d71a74-dockercfg-2jbwf\""
Nov 03 21:26:47.096052 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.096028    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr"]
Nov 03 21:26:47.133323 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.133293    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkncd\" (UniqueName: \"kubernetes.io/projected/ce8d8f89-c688-4928-8ea7-ec6b22a2c74e-kube-api-access-hkncd\") pod \"payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr\" (UID: \"ce8d8f89-c688-4928-8ea7-ec6b22a2c74e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr"
Nov 03 21:26:47.133508 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.133407    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8d8f89-c688-4928-8ea7-ec6b22a2c74e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr\" (UID: \"ce8d8f89-c688-4928-8ea7-ec6b22a2c74e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr"
Nov 03 21:26:47.234639 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.234604    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkncd\" (UniqueName: \"kubernetes.io/projected/ce8d8f89-c688-4928-8ea7-ec6b22a2c74e-kube-api-access-hkncd\") pod \"payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr\" (UID: \"ce8d8f89-c688-4928-8ea7-ec6b22a2c74e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr"
Nov 03 21:26:47.234794 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.234708    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8d8f89-c688-4928-8ea7-ec6b22a2c74e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr\" (UID: \"ce8d8f89-c688-4928-8ea7-ec6b22a2c74e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr"
Nov 03 21:26:47.235475 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.235457    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8d8f89-c688-4928-8ea7-ec6b22a2c74e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr\" (UID: \"ce8d8f89-c688-4928-8ea7-ec6b22a2c74e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr"
Nov 03 21:26:47.293911 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.293834    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkncd\" (UniqueName: \"kubernetes.io/projected/ce8d8f89-c688-4928-8ea7-ec6b22a2c74e-kube-api-access-hkncd\") pod \"payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr\" (UID: \"ce8d8f89-c688-4928-8ea7-ec6b22a2c74e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr"
Nov 03 21:26:47.342811 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.342783    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7"
Nov 03 21:26:47.351830 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.351804    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr"
Nov 03 21:26:47.436962 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.436932    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42e8cb6e-820e-45d6-8af3-fa981f2bfa7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e8cb6e-820e-45d6-8af3-fa981f2bfa7a-plugins-config-volume\") pod \"42e8cb6e-820e-45d6-8af3-fa981f2bfa7a\" (UID: \"42e8cb6e-820e-45d6-8af3-fa981f2bfa7a\") "
Nov 03 21:26:47.437083 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.437031    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42e8cb6e-820e-45d6-8af3-fa981f2bfa7a-kube-api-access-tnjsm\" (UniqueName: \"kubernetes.io/projected/42e8cb6e-820e-45d6-8af3-fa981f2bfa7a-kube-api-access-tnjsm\") pod \"42e8cb6e-820e-45d6-8af3-fa981f2bfa7a\" (UID: \"42e8cb6e-820e-45d6-8af3-fa981f2bfa7a\") "
Nov 03 21:26:47.437335 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.437310    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42e8cb6e-820e-45d6-8af3-fa981f2bfa7a-plugins-config-volume" pod "42e8cb6e-820e-45d6-8af3-fa981f2bfa7a" (UID: "42e8cb6e-820e-45d6-8af3-fa981f2bfa7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:47.439144 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.439122    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e8cb6e-820e-45d6-8af3-fa981f2bfa7a-kube-api-access-tnjsm" pod "42e8cb6e-820e-45d6-8af3-fa981f2bfa7a" (UID: "42e8cb6e-820e-45d6-8af3-fa981f2bfa7a"). InnerVolumeSpecName "kube-api-access-tnjsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:47.538492 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.538456    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e8cb6e-820e-45d6-8af3-fa981f2bfa7a-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:47.538492 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.538485    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnjsm\" (UniqueName: \"kubernetes.io/projected/42e8cb6e-820e-45d6-8af3-fa981f2bfa7a-kube-api-access-tnjsm\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:47.563317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.563215    5053 generic.go:356] "Generic (PLEG): container finished" podID="42e8cb6e-820e-45d6-8af3-fa981f2bfa7a" containerID="ddebab7a55a532c0ee1a1b4712bbc6ce4445b253d308d3950b84e7e64b2a1a17" exitCode=0
Nov 03 21:26:47.563317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.563273    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7"
Nov 03 21:26:47.563317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.563294    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7" event={"ID":"42e8cb6e-820e-45d6-8af3-fa981f2bfa7a","Type":"ContainerDied","Data":"ddebab7a55a532c0ee1a1b4712bbc6ce4445b253d308d3950b84e7e64b2a1a17"}
Nov 03 21:26:47.563552 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.563345    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7" event={"ID":"42e8cb6e-820e-45d6-8af3-fa981f2bfa7a","Type":"ContainerDied","Data":"b7c510734ba33050bcca20e170f8d581d389c34b08397d13bf9b2571c8f206ff"}
Nov 03 21:26:47.563552 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.563366    5053 scope.go:122] "RemoveContainer" containerID="ddebab7a55a532c0ee1a1b4712bbc6ce4445b253d308d3950b84e7e64b2a1a17"
Nov 03 21:26:47.565231 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.565210    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv" event={"ID":"0a1fdc06-686e-499e-9c23-ae901773d3b6","Type":"ContainerStarted","Data":"83a33949ac671af1b3a7c54eaac732d1b65b39f42b70dd1145e73e94809fdd63"}
Nov 03 21:26:47.566531 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.566510    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv" event={"ID":"0a1fdc06-686e-499e-9c23-ae901773d3b6","Type":"ContainerStarted","Data":"2a81d1e779df173196eeb68fbc09984220e4f3e50a1899a86659144007c04930"}
Nov 03 21:26:47.572739 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.572724    5053 scope.go:122] "RemoveContainer" containerID="ddebab7a55a532c0ee1a1b4712bbc6ce4445b253d308d3950b84e7e64b2a1a17"
Nov 03 21:26:47.573010 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:26:47.572990    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddebab7a55a532c0ee1a1b4712bbc6ce4445b253d308d3950b84e7e64b2a1a17\": container with ID starting with ddebab7a55a532c0ee1a1b4712bbc6ce4445b253d308d3950b84e7e64b2a1a17 not found: ID does not exist" containerID="ddebab7a55a532c0ee1a1b4712bbc6ce4445b253d308d3950b84e7e64b2a1a17"
Nov 03 21:26:47.573054 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.573018    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddebab7a55a532c0ee1a1b4712bbc6ce4445b253d308d3950b84e7e64b2a1a17"} err="failed to get container status \"ddebab7a55a532c0ee1a1b4712bbc6ce4445b253d308d3950b84e7e64b2a1a17\": rpc error: code = NotFound desc = could not find container \"ddebab7a55a532c0ee1a1b4712bbc6ce4445b253d308d3950b84e7e64b2a1a17\": container with ID starting with ddebab7a55a532c0ee1a1b4712bbc6ce4445b253d308d3950b84e7e64b2a1a17 not found: ID does not exist"
Nov 03 21:26:47.724512 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.724478    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr"]
Nov 03 21:26:47.726088 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:26:47.726063    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce8d8f89_c688_4928_8ea7_ec6b22a2c74e.slice/crio-7473c4c1a7d12a8e8fb20b72cea32310a3a402cd3a09ef0c4f9c8d14295642ec WatchSource:0}: Error finding container 7473c4c1a7d12a8e8fb20b72cea32310a3a402cd3a09ef0c4f9c8d14295642ec: Status 404 returned error can't find the container with id 7473c4c1a7d12a8e8fb20b72cea32310a3a402cd3a09ef0c4f9c8d14295642ec
Nov 03 21:26:47.733080 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.733039    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv" podStartSLOduration=2.733024392 podStartE2EDuration="2.733024392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:47.729761651 +0000 UTC m=+722.223683538" watchObservedRunningTime="2026-11-03 21:26:47.733024392 +0000 UTC m=+722.226946268"
Nov 03 21:26:47.863424 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.863333    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7"]
Nov 03 21:26:47.868854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:47.868829    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-35257bc0-77f9b6487f-hgxm7"]
Nov 03 21:26:48.260993 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:48.260966    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:48.266392 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:48.266369    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:48.572368 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:48.572332    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr" event={"ID":"ce8d8f89-c688-4928-8ea7-ec6b22a2c74e","Type":"ContainerStarted","Data":"e4939359ac90512ce62e3059a98569684725aebb2f8d3fba23c1fbffbf111797"}
Nov 03 21:26:48.572544 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:48.572388    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr" event={"ID":"ce8d8f89-c688-4928-8ea7-ec6b22a2c74e","Type":"ContainerStarted","Data":"7473c4c1a7d12a8e8fb20b72cea32310a3a402cd3a09ef0c4f9c8d14295642ec"}
Nov 03 21:26:48.572544 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:48.572438    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr"
Nov 03 21:26:48.573532 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:48.573514    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:48.574524 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:48.574508    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:26:48.686319 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:48.686265    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr" podStartSLOduration=2.686251134 podStartE2EDuration="2.686251134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:48.685541444 +0000 UTC m=+723.179463320" watchObservedRunningTime="2026-11-03 21:26:48.686251134 +0000 UTC m=+723.180173000"
Nov 03 21:26:49.666591 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.666561    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42e8cb6e-820e-45d6-8af3-fa981f2bfa7a" path="/var/lib/kubelet/pods/42e8cb6e-820e-45d6-8af3-fa981f2bfa7a/volumes"
Nov 03 21:26:49.793004 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.792942    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:26:49.870532 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.870498    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istiod-ca-cert\") pod \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") "
Nov 03 21:26:49.870715 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.870571    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-workload-certs\") pod \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") "
Nov 03 21:26:49.870715 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.870679    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-workload-socket\") pod \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") "
Nov 03 21:26:49.870715 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.870710    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-envoy\") pod \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") "
Nov 03 21:26:49.870882 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.870764    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-data\") pod \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") "
Nov 03 21:26:49.870882 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.870794    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-kube-api-access-b85dz\" (UniqueName: \"kubernetes.io/projected/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-kube-api-access-b85dz\") pod \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") "
Nov 03 21:26:49.870882 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.870840    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-workload-certs" pod "0013fad5-118c-4e7b-b51f-6fdb78d47bb4" (UID: "0013fad5-118c-4e7b-b51f-6fdb78d47bb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:26:49.870882 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.870857    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istiod-ca-cert" pod "0013fad5-118c-4e7b-b51f-6fdb78d47bb4" (UID: "0013fad5-118c-4e7b-b51f-6fdb78d47bb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:49.871069 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.870874    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-credential-socket\") pod \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") "
Nov 03 21:26:49.871069 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.870948    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-workload-socket" pod "0013fad5-118c-4e7b-b51f-6fdb78d47bb4" (UID: "0013fad5-118c-4e7b-b51f-6fdb78d47bb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:26:49.871069 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.871055    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-data" pod "0013fad5-118c-4e7b-b51f-6fdb78d47bb4" (UID: "0013fad5-118c-4e7b-b51f-6fdb78d47bb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:26:49.871069 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.871061    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-podinfo\") pod \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") "
Nov 03 21:26:49.871270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.871095    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-credential-socket" pod "0013fad5-118c-4e7b-b51f-6fdb78d47bb4" (UID: "0013fad5-118c-4e7b-b51f-6fdb78d47bb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:26:49.871270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.871114    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-token\" (UniqueName: \"kubernetes.io/projected/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-token\") pod \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\" (UID: \"0013fad5-118c-4e7b-b51f-6fdb78d47bb4\") "
Nov 03 21:26:49.871569 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.871545    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:49.871691 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.871572    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:49.871691 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.871588    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:49.871691 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.871602    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:49.871691 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.871614    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:49.873273 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.873221    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-podinfo" pod "0013fad5-118c-4e7b-b51f-6fdb78d47bb4" (UID: "0013fad5-118c-4e7b-b51f-6fdb78d47bb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:26:49.873477 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.873445    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-token" pod "0013fad5-118c-4e7b-b51f-6fdb78d47bb4" (UID: "0013fad5-118c-4e7b-b51f-6fdb78d47bb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:49.873554 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.873532    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-envoy" pod "0013fad5-118c-4e7b-b51f-6fdb78d47bb4" (UID: "0013fad5-118c-4e7b-b51f-6fdb78d47bb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:26:49.873596 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.873538    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-kube-api-access-b85dz" pod "0013fad5-118c-4e7b-b51f-6fdb78d47bb4" (UID: "0013fad5-118c-4e7b-b51f-6fdb78d47bb4"). InnerVolumeSpecName "kube-api-access-b85dz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:49.939805 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.939774    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx"]
Nov 03 21:26:49.973004 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.972978    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:49.973004 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.973001    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:49.973004 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.973011    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:49.973219 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:49.973021    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b85dz\" (UniqueName: \"kubernetes.io/projected/0013fad5-118c-4e7b-b51f-6fdb78d47bb4-kube-api-access-b85dz\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:50.049630 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:50.049552    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc"]
Nov 03 21:26:50.586142 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:50.586112    5053 generic.go:356] "Generic (PLEG): container finished" podID="0013fad5-118c-4e7b-b51f-6fdb78d47bb4" containerID="bc65cfdacbd84cec9a6ca6d545cc1d151c447a0e853692de19931dec6f822430" exitCode=0
Nov 03 21:26:50.586351 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:50.586160    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"
Nov 03 21:26:50.586351 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:50.586196    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td" event={"ID":"0013fad5-118c-4e7b-b51f-6fdb78d47bb4","Type":"ContainerDied","Data":"bc65cfdacbd84cec9a6ca6d545cc1d151c447a0e853692de19931dec6f822430"}
Nov 03 21:26:50.586351 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:50.586281    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td" event={"ID":"0013fad5-118c-4e7b-b51f-6fdb78d47bb4","Type":"ContainerDied","Data":"15764b73b435eb13d2449ed5a06fe4200c862ff55cec8a054334a1796865ce4c"}
Nov 03 21:26:50.586351 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:50.586314    5053 scope.go:122] "RemoveContainer" containerID="bc65cfdacbd84cec9a6ca6d545cc1d151c447a0e853692de19931dec6f822430"
Nov 03 21:26:50.596907 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:50.596887    5053 scope.go:122] "RemoveContainer" containerID="0d93dd01f2687ba7b43c249262e903018833ca5f5e9972b53d2d9825f13e4119"
Nov 03 21:26:50.607020 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:50.606993    5053 scope.go:122] "RemoveContainer" containerID="bc65cfdacbd84cec9a6ca6d545cc1d151c447a0e853692de19931dec6f822430"
Nov 03 21:26:50.607349 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:26:50.607323    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc65cfdacbd84cec9a6ca6d545cc1d151c447a0e853692de19931dec6f822430\": container with ID starting with bc65cfdacbd84cec9a6ca6d545cc1d151c447a0e853692de19931dec6f822430 not found: ID does not exist" containerID="bc65cfdacbd84cec9a6ca6d545cc1d151c447a0e853692de19931dec6f822430"
Nov 03 21:26:50.607428 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:50.607361    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc65cfdacbd84cec9a6ca6d545cc1d151c447a0e853692de19931dec6f822430"} err="failed to get container status \"bc65cfdacbd84cec9a6ca6d545cc1d151c447a0e853692de19931dec6f822430\": rpc error: code = NotFound desc = could not find container \"bc65cfdacbd84cec9a6ca6d545cc1d151c447a0e853692de19931dec6f822430\": container with ID starting with bc65cfdacbd84cec9a6ca6d545cc1d151c447a0e853692de19931dec6f822430 not found: ID does not exist"
Nov 03 21:26:50.607428 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:50.607388    5053 scope.go:122] "RemoveContainer" containerID="0d93dd01f2687ba7b43c249262e903018833ca5f5e9972b53d2d9825f13e4119"
Nov 03 21:26:50.607660 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:26:50.607640    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d93dd01f2687ba7b43c249262e903018833ca5f5e9972b53d2d9825f13e4119\": container with ID starting with 0d93dd01f2687ba7b43c249262e903018833ca5f5e9972b53d2d9825f13e4119 not found: ID does not exist" containerID="0d93dd01f2687ba7b43c249262e903018833ca5f5e9972b53d2d9825f13e4119"
Nov 03 21:26:50.607729 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:50.607667    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d93dd01f2687ba7b43c249262e903018833ca5f5e9972b53d2d9825f13e4119"} err="failed to get container status \"0d93dd01f2687ba7b43c249262e903018833ca5f5e9972b53d2d9825f13e4119\": rpc error: code = NotFound desc = could not find container \"0d93dd01f2687ba7b43c249262e903018833ca5f5e9972b53d2d9825f13e4119\": container with ID starting with 0d93dd01f2687ba7b43c249262e903018833ca5f5e9972b53d2d9825f13e4119 not found: ID does not exist"
Nov 03 21:26:50.748205 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:50.748172    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"]
Nov 03 21:26:50.751637 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:50.751609    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e7203b81-gw-openshift-default-76d48446bf-956td"]
Nov 03 21:26:51.670010 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:51.669972    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0013fad5-118c-4e7b-b51f-6fdb78d47bb4" path="/var/lib/kubelet/pods/0013fad5-118c-4e7b-b51f-6fdb78d47bb4/volumes"
Nov 03 21:26:54.940953 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:54.940886    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx" podUID="8d5c416d-5668-4718-965e-e5a2b5e0ff5c" containerName="payload-processing" containerID="cri-o://6fcf3f9f1bf86cffb0ed0b894de4de3f8c199ebead73974d1ce90741de492409" gracePeriod=25
Nov 03 21:26:55.050224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.050143    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc" podUID="6d4a1b84-b85a-4ff9-b2ab-31d651df7059" containerName="payload-pre-processing" containerID="cri-o://eb1125cac2ec0abc9253cecde3b7eceb9fde6a1a85306cc7d492073c31002509" gracePeriod=25
Nov 03 21:26:55.209141 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.209112    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx"
Nov 03 21:26:55.318424 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.318399    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc"
Nov 03 21:26:55.329362 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.329337    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d5c416d-5668-4718-965e-e5a2b5e0ff5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5c416d-5668-4718-965e-e5a2b5e0ff5c-plugins-config-volume\") pod \"8d5c416d-5668-4718-965e-e5a2b5e0ff5c\" (UID: \"8d5c416d-5668-4718-965e-e5a2b5e0ff5c\") "
Nov 03 21:26:55.329518 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.329434    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d5c416d-5668-4718-965e-e5a2b5e0ff5c-kube-api-access-2rc4n\" (UniqueName: \"kubernetes.io/projected/8d5c416d-5668-4718-965e-e5a2b5e0ff5c-kube-api-access-2rc4n\") pod \"8d5c416d-5668-4718-965e-e5a2b5e0ff5c\" (UID: \"8d5c416d-5668-4718-965e-e5a2b5e0ff5c\") "
Nov 03 21:26:55.329748 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.329723    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5c416d-5668-4718-965e-e5a2b5e0ff5c-plugins-config-volume" pod "8d5c416d-5668-4718-965e-e5a2b5e0ff5c" (UID: "8d5c416d-5668-4718-965e-e5a2b5e0ff5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:55.331800 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.331771    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d5c416d-5668-4718-965e-e5a2b5e0ff5c-kube-api-access-2rc4n" pod "8d5c416d-5668-4718-965e-e5a2b5e0ff5c" (UID: "8d5c416d-5668-4718-965e-e5a2b5e0ff5c"). InnerVolumeSpecName "kube-api-access-2rc4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:55.430471 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.430438    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d4a1b84-b85a-4ff9-b2ab-31d651df7059-kube-api-access-mbdf9\" (UniqueName: \"kubernetes.io/projected/6d4a1b84-b85a-4ff9-b2ab-31d651df7059-kube-api-access-mbdf9\") pod \"6d4a1b84-b85a-4ff9-b2ab-31d651df7059\" (UID: \"6d4a1b84-b85a-4ff9-b2ab-31d651df7059\") "
Nov 03 21:26:55.430643 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.430537    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d4a1b84-b85a-4ff9-b2ab-31d651df7059-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d4a1b84-b85a-4ff9-b2ab-31d651df7059-plugins-config-volume\") pod \"6d4a1b84-b85a-4ff9-b2ab-31d651df7059\" (UID: \"6d4a1b84-b85a-4ff9-b2ab-31d651df7059\") "
Nov 03 21:26:55.430760 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.430747    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5c416d-5668-4718-965e-e5a2b5e0ff5c-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:55.430760 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.430760    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rc4n\" (UniqueName: \"kubernetes.io/projected/8d5c416d-5668-4718-965e-e5a2b5e0ff5c-kube-api-access-2rc4n\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:55.430927 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.430904    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d4a1b84-b85a-4ff9-b2ab-31d651df7059-plugins-config-volume" pod "6d4a1b84-b85a-4ff9-b2ab-31d651df7059" (UID: "6d4a1b84-b85a-4ff9-b2ab-31d651df7059"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:55.432717 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.432695    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d4a1b84-b85a-4ff9-b2ab-31d651df7059-kube-api-access-mbdf9" pod "6d4a1b84-b85a-4ff9-b2ab-31d651df7059" (UID: "6d4a1b84-b85a-4ff9-b2ab-31d651df7059"). InnerVolumeSpecName "kube-api-access-mbdf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:55.531697 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.531610    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbdf9\" (UniqueName: \"kubernetes.io/projected/6d4a1b84-b85a-4ff9-b2ab-31d651df7059-kube-api-access-mbdf9\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:55.531697 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.531638    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d4a1b84-b85a-4ff9-b2ab-31d651df7059-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:55.618329 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.618297    5053 generic.go:356] "Generic (PLEG): container finished" podID="6d4a1b84-b85a-4ff9-b2ab-31d651df7059" containerID="eb1125cac2ec0abc9253cecde3b7eceb9fde6a1a85306cc7d492073c31002509" exitCode=0
Nov 03 21:26:55.618502 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.618348    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc"
Nov 03 21:26:55.618502 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.618377    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc" event={"ID":"6d4a1b84-b85a-4ff9-b2ab-31d651df7059","Type":"ContainerDied","Data":"eb1125cac2ec0abc9253cecde3b7eceb9fde6a1a85306cc7d492073c31002509"}
Nov 03 21:26:55.618502 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.618436    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc" event={"ID":"6d4a1b84-b85a-4ff9-b2ab-31d651df7059","Type":"ContainerDied","Data":"c1fc337b6758d6dee7cd74f33fbaa2463e2e69d39aa980c76b7a9dd086f5dcbf"}
Nov 03 21:26:55.618502 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.618467    5053 scope.go:122] "RemoveContainer" containerID="eb1125cac2ec0abc9253cecde3b7eceb9fde6a1a85306cc7d492073c31002509"
Nov 03 21:26:55.620023 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.620001    5053 generic.go:356] "Generic (PLEG): container finished" podID="8d5c416d-5668-4718-965e-e5a2b5e0ff5c" containerID="6fcf3f9f1bf86cffb0ed0b894de4de3f8c199ebead73974d1ce90741de492409" exitCode=0
Nov 03 21:26:55.620144 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.620063    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx"
Nov 03 21:26:55.620144 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.620080    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx" event={"ID":"8d5c416d-5668-4718-965e-e5a2b5e0ff5c","Type":"ContainerDied","Data":"6fcf3f9f1bf86cffb0ed0b894de4de3f8c199ebead73974d1ce90741de492409"}
Nov 03 21:26:55.620144 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.620116    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx" event={"ID":"8d5c416d-5668-4718-965e-e5a2b5e0ff5c","Type":"ContainerDied","Data":"4bb088591bb80720c51441012cffa5a5b9d38defb5df7d575337c9b6da7970a8"}
Nov 03 21:26:55.640552 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.640449    5053 scope.go:122] "RemoveContainer" containerID="eb1125cac2ec0abc9253cecde3b7eceb9fde6a1a85306cc7d492073c31002509"
Nov 03 21:26:55.640954 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:26:55.640920    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb1125cac2ec0abc9253cecde3b7eceb9fde6a1a85306cc7d492073c31002509\": container with ID starting with eb1125cac2ec0abc9253cecde3b7eceb9fde6a1a85306cc7d492073c31002509 not found: ID does not exist" containerID="eb1125cac2ec0abc9253cecde3b7eceb9fde6a1a85306cc7d492073c31002509"
Nov 03 21:26:55.641050 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.640965    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb1125cac2ec0abc9253cecde3b7eceb9fde6a1a85306cc7d492073c31002509"} err="failed to get container status \"eb1125cac2ec0abc9253cecde3b7eceb9fde6a1a85306cc7d492073c31002509\": rpc error: code = NotFound desc = could not find container \"eb1125cac2ec0abc9253cecde3b7eceb9fde6a1a85306cc7d492073c31002509\": container with ID starting with eb1125cac2ec0abc9253cecde3b7eceb9fde6a1a85306cc7d492073c31002509 not found: ID does not exist"
Nov 03 21:26:55.641050 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.640991    5053 scope.go:122] "RemoveContainer" containerID="6fcf3f9f1bf86cffb0ed0b894de4de3f8c199ebead73974d1ce90741de492409"
Nov 03 21:26:55.651829 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.651803    5053 scope.go:122] "RemoveContainer" containerID="6fcf3f9f1bf86cffb0ed0b894de4de3f8c199ebead73974d1ce90741de492409"
Nov 03 21:26:55.652162 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:26:55.652141    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fcf3f9f1bf86cffb0ed0b894de4de3f8c199ebead73974d1ce90741de492409\": container with ID starting with 6fcf3f9f1bf86cffb0ed0b894de4de3f8c199ebead73974d1ce90741de492409 not found: ID does not exist" containerID="6fcf3f9f1bf86cffb0ed0b894de4de3f8c199ebead73974d1ce90741de492409"
Nov 03 21:26:55.652348 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.652170    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fcf3f9f1bf86cffb0ed0b894de4de3f8c199ebead73974d1ce90741de492409"} err="failed to get container status \"6fcf3f9f1bf86cffb0ed0b894de4de3f8c199ebead73974d1ce90741de492409\": rpc error: code = NotFound desc = could not find container \"6fcf3f9f1bf86cffb0ed0b894de4de3f8c199ebead73974d1ce90741de492409\": container with ID starting with 6fcf3f9f1bf86cffb0ed0b894de4de3f8c199ebead73974d1ce90741de492409 not found: ID does not exist"
Nov 03 21:26:55.781564 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.781532    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx"]
Nov 03 21:26:55.786464 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.786389    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-daf21c29-64b9d776d8-kcmfx"]
Nov 03 21:26:55.892749 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.892720    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc"]
Nov 03 21:26:55.896467 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:55.896441    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-daf21c29-7559b4cb4c-jchxc"]
Nov 03 21:26:56.765556 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.765518    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h"]
Nov 03 21:26:56.766282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.766215    5053 cpu_manager.go:414] "containerMap: removing container" podUID="42e8cb6e-820e-45d6-8af3-fa981f2bfa7a" containerName="payload-pre-processing"
Nov 03 21:26:56.766282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.766255    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42e8cb6e-820e-45d6-8af3-fa981f2bfa7a" containerName="payload-pre-processing"
Nov 03 21:26:56.766282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.766270    5053 cpu_manager.go:414] "containerMap: removing container" podUID="8d5c416d-5668-4718-965e-e5a2b5e0ff5c" containerName="payload-processing"
Nov 03 21:26:56.766282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.766278    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d5c416d-5668-4718-965e-e5a2b5e0ff5c" containerName="payload-processing"
Nov 03 21:26:56.766467 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.766324    5053 cpu_manager.go:414] "containerMap: removing container" podUID="0013fad5-118c-4e7b-b51f-6fdb78d47bb4" containerName="istio-proxy"
Nov 03 21:26:56.766467 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.766331    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0013fad5-118c-4e7b-b51f-6fdb78d47bb4" containerName="istio-proxy"
Nov 03 21:26:56.766467 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.766341    5053 cpu_manager.go:414] "containerMap: removing container" podUID="0013fad5-118c-4e7b-b51f-6fdb78d47bb4" containerName="istio-proxy"
Nov 03 21:26:56.766467 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.766348    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0013fad5-118c-4e7b-b51f-6fdb78d47bb4" containerName="istio-proxy"
Nov 03 21:26:56.766467 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.766357    5053 cpu_manager.go:414] "containerMap: removing container" podUID="6d4a1b84-b85a-4ff9-b2ab-31d651df7059" containerName="payload-pre-processing"
Nov 03 21:26:56.766467 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.766363    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d4a1b84-b85a-4ff9-b2ab-31d651df7059" containerName="payload-pre-processing"
Nov 03 21:26:56.766640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.766487    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="0013fad5-118c-4e7b-b51f-6fdb78d47bb4" containerName="istio-proxy"
Nov 03 21:26:56.766640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.766497    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="0013fad5-118c-4e7b-b51f-6fdb78d47bb4" containerName="istio-proxy"
Nov 03 21:26:56.766640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.766504    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d4a1b84-b85a-4ff9-b2ab-31d651df7059" containerName="payload-pre-processing"
Nov 03 21:26:56.766640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.766517    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="42e8cb6e-820e-45d6-8af3-fa981f2bfa7a" containerName="payload-pre-processing"
Nov 03 21:26:56.766640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.766525    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d5c416d-5668-4718-965e-e5a2b5e0ff5c" containerName="payload-processing"
Nov 03 21:26:56.771827 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.771805    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h"
Nov 03 21:26:56.824616 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.824586    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h"]
Nov 03 21:26:56.871961 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.871302    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-da5f0b-dockercfg-png28\""
Nov 03 21:26:56.871961 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.871499    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-da5f0b\""
Nov 03 21:26:56.923420 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.923382    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt"]
Nov 03 21:26:56.927634 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.927607    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt"
Nov 03 21:26:56.946294 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.946261    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77tjr\" (UniqueName: \"kubernetes.io/projected/40e7f3d6-f0f4-456e-864e-1c3224bb3463-kube-api-access-77tjr\") pod \"payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h\" (UID: \"40e7f3d6-f0f4-456e-864e-1c3224bb3463\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h"
Nov 03 21:26:56.946389 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.946318    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e7f3d6-f0f4-456e-864e-1c3224bb3463-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h\" (UID: \"40e7f3d6-f0f4-456e-864e-1c3224bb3463\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h"
Nov 03 21:26:56.976768 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:56.976382    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt"]
Nov 03 21:26:57.047581 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:57.047492    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77tjr\" (UniqueName: \"kubernetes.io/projected/40e7f3d6-f0f4-456e-864e-1c3224bb3463-kube-api-access-77tjr\") pod \"payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h\" (UID: \"40e7f3d6-f0f4-456e-864e-1c3224bb3463\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h"
Nov 03 21:26:57.047581 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:57.047548    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e7f3d6-f0f4-456e-864e-1c3224bb3463-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h\" (UID: \"40e7f3d6-f0f4-456e-864e-1c3224bb3463\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h"
Nov 03 21:26:57.047802 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:57.047596    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/770bdb59-84e0-449d-aaea-d75986b816b1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt\" (UID: \"770bdb59-84e0-449d-aaea-d75986b816b1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt"
Nov 03 21:26:57.047802 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:57.047624    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rht6m\" (UniqueName: \"kubernetes.io/projected/770bdb59-84e0-449d-aaea-d75986b816b1-kube-api-access-rht6m\") pod \"payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt\" (UID: \"770bdb59-84e0-449d-aaea-d75986b816b1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt"
Nov 03 21:26:57.048345 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:57.048326    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e7f3d6-f0f4-456e-864e-1c3224bb3463-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h\" (UID: \"40e7f3d6-f0f4-456e-864e-1c3224bb3463\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h"
Nov 03 21:26:57.104488 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:57.104455    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77tjr\" (UniqueName: \"kubernetes.io/projected/40e7f3d6-f0f4-456e-864e-1c3224bb3463-kube-api-access-77tjr\") pod \"payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h\" (UID: \"40e7f3d6-f0f4-456e-864e-1c3224bb3463\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h"
Nov 03 21:26:57.148294 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:57.148263    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/770bdb59-84e0-449d-aaea-d75986b816b1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt\" (UID: \"770bdb59-84e0-449d-aaea-d75986b816b1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt"
Nov 03 21:26:57.148294 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:57.148300    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rht6m\" (UniqueName: \"kubernetes.io/projected/770bdb59-84e0-449d-aaea-d75986b816b1-kube-api-access-rht6m\") pod \"payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt\" (UID: \"770bdb59-84e0-449d-aaea-d75986b816b1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt"
Nov 03 21:26:57.148976 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:57.148949    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/770bdb59-84e0-449d-aaea-d75986b816b1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt\" (UID: \"770bdb59-84e0-449d-aaea-d75986b816b1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt"
Nov 03 21:26:57.204852 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:57.204819    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rht6m\" (UniqueName: \"kubernetes.io/projected/770bdb59-84e0-449d-aaea-d75986b816b1-kube-api-access-rht6m\") pod \"payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt\" (UID: \"770bdb59-84e0-449d-aaea-d75986b816b1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt"
Nov 03 21:26:57.240989 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:57.240949    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt"
Nov 03 21:26:57.385193 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:57.385160    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h"
Nov 03 21:26:57.435291 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:57.435254    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs"]
Nov 03 21:26:57.438676 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:57.438652    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35257bc0-jpvzs"]
Nov 03 21:26:57.618339 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:57.618316    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt"]
Nov 03 21:26:57.619732 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:26:57.619706    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770bdb59_84e0_449d_aaea_d75986b816b1.slice/crio-9151e04fafe2c7c948ae859e094da167c3773413aaba9fadfa4bb093ca8503d5 WatchSource:0}: Error finding container 9151e04fafe2c7c948ae859e094da167c3773413aaba9fadfa4bb093ca8503d5: Status 404 returned error can't find the container with id 9151e04fafe2c7c948ae859e094da167c3773413aaba9fadfa4bb093ca8503d5
Nov 03 21:26:57.641326 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:57.641286    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt" event={"ID":"770bdb59-84e0-449d-aaea-d75986b816b1","Type":"ContainerStarted","Data":"9151e04fafe2c7c948ae859e094da167c3773413aaba9fadfa4bb093ca8503d5"}
Nov 03 21:26:57.665954 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:57.665919    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d4a1b84-b85a-4ff9-b2ab-31d651df7059" path="/var/lib/kubelet/pods/6d4a1b84-b85a-4ff9-b2ab-31d651df7059/volumes"
Nov 03 21:26:57.666479 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:57.666456    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fa768a1-1de5-4124-a635-605562ef0ab4" path="/var/lib/kubelet/pods/7fa768a1-1de5-4124-a635-605562ef0ab4/volumes"
Nov 03 21:26:57.666881 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:57.666866    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d5c416d-5668-4718-965e-e5a2b5e0ff5c" path="/var/lib/kubelet/pods/8d5c416d-5668-4718-965e-e5a2b5e0ff5c/volumes"
Nov 03 21:26:57.758129 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:57.758101    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h"]
Nov 03 21:26:57.759743 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:26:57.759708    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e7f3d6_f0f4_456e_864e_1c3224bb3463.slice/crio-42ea98cb96b5fea5608decc81744f75385af5d7fa9f5dbbeabf5b25d5e2441fe WatchSource:0}: Error finding container 42ea98cb96b5fea5608decc81744f75385af5d7fa9f5dbbeabf5b25d5e2441fe: Status 404 returned error can't find the container with id 42ea98cb96b5fea5608decc81744f75385af5d7fa9f5dbbeabf5b25d5e2441fe
Nov 03 21:26:58.649639 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:58.649597    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt" event={"ID":"770bdb59-84e0-449d-aaea-d75986b816b1","Type":"ContainerStarted","Data":"d4b5ee122224e6d49d9925cc73b552008c719aa2003eaa1ca48afdf233d32bae"}
Nov 03 21:26:58.650223 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:58.649702    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt"
Nov 03 21:26:58.651574 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:58.651550    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h" event={"ID":"40e7f3d6-f0f4-456e-864e-1c3224bb3463","Type":"ContainerStarted","Data":"f5b5c9e2ddfec1c0e0aa0ac52337d096136207eb50c0fbfccfc672afa1e8a28a"}
Nov 03 21:26:58.651717 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:58.651594    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h" event={"ID":"40e7f3d6-f0f4-456e-864e-1c3224bb3463","Type":"ContainerStarted","Data":"42ea98cb96b5fea5608decc81744f75385af5d7fa9f5dbbeabf5b25d5e2441fe"}
Nov 03 21:26:58.651717 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:58.651663    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h"
Nov 03 21:26:58.763760 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:58.763709    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt" podStartSLOduration=2.7636959709999998 podStartE2EDuration="2.763695971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:58.762228496 +0000 UTC m=+733.256150372" watchObservedRunningTime="2026-11-03 21:26:58.763695971 +0000 UTC m=+733.257617845"
Nov 03 21:26:58.876760 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:58.876591    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h" podStartSLOduration=2.876570784 podStartE2EDuration="2.876570784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:58.874521184 +0000 UTC m=+733.368443085" watchObservedRunningTime="2026-11-03 21:26:58.876570784 +0000 UTC m=+733.370492662"
Nov 03 21:26:59.072548 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:59.072515    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"]
Nov 03 21:26:59.072766 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:59.072742    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph" podUID="610101d0-cf94-41b6-a21e-18e652eb9b03" containerName="istio-proxy" containerID="cri-o://0eb784963ed15b7ac5e8c8e38aaaacba01fbee65657be7808a90290f90b1b648" gracePeriod=30
Nov 03 21:26:59.581793 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:26:59.581764    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr"
Nov 03 21:27:00.492650 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.492616    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"]
Nov 03 21:27:00.496439 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.496417    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.572147 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.572107    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-6de8e6-openshift-default-dockercfg-vxss9\""
Nov 03 21:27:00.589452 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.589419    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.589629 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.589464    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d17433-dc86-491b-9276-080c84dfdc8e-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.589629 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.589489    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.589713 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.589682    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.589753 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.589712    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d17433-dc86-491b-9276-080c84dfdc8e-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.589808 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.589751    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.589860 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.589809    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clm7n\" (UniqueName: \"kubernetes.io/projected/51d17433-dc86-491b-9276-080c84dfdc8e-kube-api-access-clm7n\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.589860 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.589835    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d17433-dc86-491b-9276-080c84dfdc8e-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.589950 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.589865    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.613370 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.613334    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"]
Nov 03 21:27:00.691407 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.691372    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.691608 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.691441    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clm7n\" (UniqueName: \"kubernetes.io/projected/51d17433-dc86-491b-9276-080c84dfdc8e-kube-api-access-clm7n\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.691608 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.691473    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d17433-dc86-491b-9276-080c84dfdc8e-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.691608 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.691502    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.691608 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.691551    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.691608 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.691592    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d17433-dc86-491b-9276-080c84dfdc8e-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.691875 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.691633    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.691875 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.691735    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.691875 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.691762    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d17433-dc86-491b-9276-080c84dfdc8e-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.692018 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.691869    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.692179 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.692154    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.692338 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.692178    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.692338 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.692318    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.692620 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.692599    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d17433-dc86-491b-9276-080c84dfdc8e-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.694339 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.694300    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.695085 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.695067    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d17433-dc86-491b-9276-080c84dfdc8e-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.748645 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.748573    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clm7n\" (UniqueName: \"kubernetes.io/projected/51d17433-dc86-491b-9276-080c84dfdc8e-kube-api-access-clm7n\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.748645 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.748576    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d17433-dc86-491b-9276-080c84dfdc8e-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:00.809437 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:00.809403    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:01.240098 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:01.240062    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"]
Nov 03 21:27:01.279975 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:27:01.279946    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d17433_dc86_491b_9276_080c84dfdc8e.slice/crio-38376f3c5fd678a2115b066b4a967ee7c2a08b813511415fa5eb599364b7c995 WatchSource:0}: Error finding container 38376f3c5fd678a2115b066b4a967ee7c2a08b813511415fa5eb599364b7c995: Status 404 returned error can't find the container with id 38376f3c5fd678a2115b066b4a967ee7c2a08b813511415fa5eb599364b7c995
Nov 03 21:27:01.282109 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:01.282074    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:27:01.282180 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:01.282166    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:27:01.282225 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:01.282195    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:27:01.673477 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:01.673379    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd" event={"ID":"51d17433-dc86-491b-9276-080c84dfdc8e","Type":"ContainerStarted","Data":"77cc02f8698ec91ff89cc2168890c6970d6b9d597d41712c8ef537f3644d6f3f"}
Nov 03 21:27:01.673477 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:01.673432    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd" event={"ID":"51d17433-dc86-491b-9276-080c84dfdc8e","Type":"ContainerStarted","Data":"38376f3c5fd678a2115b066b4a967ee7c2a08b813511415fa5eb599364b7c995"}
Nov 03 21:27:01.789123 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:01.789067    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd" podStartSLOduration=1.789051902 podStartE2EDuration="1.789051902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:01.787514973 +0000 UTC m=+736.281436871" watchObservedRunningTime="2026-11-03 21:27:01.789051902 +0000 UTC m=+736.282973777"
Nov 03 21:27:01.809717 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:01.809688    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:02.816549 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:02.816513    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:03.685886 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:03.685852    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:03.686731 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:03.686711    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:04.332145 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.332122    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:27:04.436859 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.436769    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/610101d0-cf94-41b6-a21e-18e652eb9b03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/610101d0-cf94-41b6-a21e-18e652eb9b03-istiod-ca-cert\") pod \"610101d0-cf94-41b6-a21e-18e652eb9b03\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") "
Nov 03 21:27:04.436859 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.436821    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-credential-socket\") pod \"610101d0-cf94-41b6-a21e-18e652eb9b03\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") "
Nov 03 21:27:04.437085 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.436867    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-podinfo\") pod \"610101d0-cf94-41b6-a21e-18e652eb9b03\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") "
Nov 03 21:27:04.437085 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.436896    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-data\") pod \"610101d0-cf94-41b6-a21e-18e652eb9b03\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") "
Nov 03 21:27:04.437085 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.436963    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/610101d0-cf94-41b6-a21e-18e652eb9b03-kube-api-access-m7pts\" (UniqueName: \"kubernetes.io/projected/610101d0-cf94-41b6-a21e-18e652eb9b03-kube-api-access-m7pts\") pod \"610101d0-cf94-41b6-a21e-18e652eb9b03\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") "
Nov 03 21:27:04.437085 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.436995    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-workload-certs\") pod \"610101d0-cf94-41b6-a21e-18e652eb9b03\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") "
Nov 03 21:27:04.437085 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.437041    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-token\" (UniqueName: \"kubernetes.io/projected/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-token\") pod \"610101d0-cf94-41b6-a21e-18e652eb9b03\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") "
Nov 03 21:27:04.437388 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.437100    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-workload-socket\") pod \"610101d0-cf94-41b6-a21e-18e652eb9b03\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") "
Nov 03 21:27:04.437388 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.437141    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-envoy\") pod \"610101d0-cf94-41b6-a21e-18e652eb9b03\" (UID: \"610101d0-cf94-41b6-a21e-18e652eb9b03\") "
Nov 03 21:27:04.437388 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.437171    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-data" pod "610101d0-cf94-41b6-a21e-18e652eb9b03" (UID: "610101d0-cf94-41b6-a21e-18e652eb9b03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:27:04.437388 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.437197    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-credential-socket" pod "610101d0-cf94-41b6-a21e-18e652eb9b03" (UID: "610101d0-cf94-41b6-a21e-18e652eb9b03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:27:04.437388 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.437352    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-workload-certs" pod "610101d0-cf94-41b6-a21e-18e652eb9b03" (UID: "610101d0-cf94-41b6-a21e-18e652eb9b03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:27:04.437610 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.437406    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-workload-socket" pod "610101d0-cf94-41b6-a21e-18e652eb9b03" (UID: "610101d0-cf94-41b6-a21e-18e652eb9b03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:27:04.437610 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.437566    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/610101d0-cf94-41b6-a21e-18e652eb9b03-istiod-ca-cert" pod "610101d0-cf94-41b6-a21e-18e652eb9b03" (UID: "610101d0-cf94-41b6-a21e-18e652eb9b03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:27:04.437610 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.437579    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:27:04.437610 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.437595    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:27:04.437610 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.437605    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:27:04.437783 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.437617    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:27:04.439400 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.439380    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-podinfo" pod "610101d0-cf94-41b6-a21e-18e652eb9b03" (UID: "610101d0-cf94-41b6-a21e-18e652eb9b03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:27:04.439633 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.439612    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-envoy" pod "610101d0-cf94-41b6-a21e-18e652eb9b03" (UID: "610101d0-cf94-41b6-a21e-18e652eb9b03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:27:04.439826 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.439801    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-token" pod "610101d0-cf94-41b6-a21e-18e652eb9b03" (UID: "610101d0-cf94-41b6-a21e-18e652eb9b03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:27:04.439826 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.439817    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610101d0-cf94-41b6-a21e-18e652eb9b03-kube-api-access-m7pts" pod "610101d0-cf94-41b6-a21e-18e652eb9b03" (UID: "610101d0-cf94-41b6-a21e-18e652eb9b03"). InnerVolumeSpecName "kube-api-access-m7pts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:27:04.538621 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.538589    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:27:04.538621 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.538616    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7pts\" (UniqueName: \"kubernetes.io/projected/610101d0-cf94-41b6-a21e-18e652eb9b03-kube-api-access-m7pts\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:27:04.538621 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.538626    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:27:04.538845 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.538635    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/610101d0-cf94-41b6-a21e-18e652eb9b03-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:27:04.538845 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.538646    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/610101d0-cf94-41b6-a21e-18e652eb9b03-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:27:04.692729 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.692648    5053 generic.go:356] "Generic (PLEG): container finished" podID="610101d0-cf94-41b6-a21e-18e652eb9b03" containerID="0eb784963ed15b7ac5e8c8e38aaaacba01fbee65657be7808a90290f90b1b648" exitCode=0
Nov 03 21:27:04.692729 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.692702    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"
Nov 03 21:27:04.692919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.692732    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph" event={"ID":"610101d0-cf94-41b6-a21e-18e652eb9b03","Type":"ContainerDied","Data":"0eb784963ed15b7ac5e8c8e38aaaacba01fbee65657be7808a90290f90b1b648"}
Nov 03 21:27:04.692919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.692787    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph" event={"ID":"610101d0-cf94-41b6-a21e-18e652eb9b03","Type":"ContainerDied","Data":"e1bcf553e7457c4909f8fee1cbad3f6c308ead0072043b740eaac5bdeb786042"}
Nov 03 21:27:04.692919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.692811    5053 scope.go:122] "RemoveContainer" containerID="0eb784963ed15b7ac5e8c8e38aaaacba01fbee65657be7808a90290f90b1b648"
Nov 03 21:27:04.704017 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.703999    5053 scope.go:122] "RemoveContainer" containerID="288fdff87ee272d32b860033ef32db18a1ff944e5830b97c229ae91d67455d9f"
Nov 03 21:27:04.713223 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.713205    5053 scope.go:122] "RemoveContainer" containerID="0eb784963ed15b7ac5e8c8e38aaaacba01fbee65657be7808a90290f90b1b648"
Nov 03 21:27:04.713500 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:27:04.713481    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eb784963ed15b7ac5e8c8e38aaaacba01fbee65657be7808a90290f90b1b648\": container with ID starting with 0eb784963ed15b7ac5e8c8e38aaaacba01fbee65657be7808a90290f90b1b648 not found: ID does not exist" containerID="0eb784963ed15b7ac5e8c8e38aaaacba01fbee65657be7808a90290f90b1b648"
Nov 03 21:27:04.713553 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.713510    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb784963ed15b7ac5e8c8e38aaaacba01fbee65657be7808a90290f90b1b648"} err="failed to get container status \"0eb784963ed15b7ac5e8c8e38aaaacba01fbee65657be7808a90290f90b1b648\": rpc error: code = NotFound desc = could not find container \"0eb784963ed15b7ac5e8c8e38aaaacba01fbee65657be7808a90290f90b1b648\": container with ID starting with 0eb784963ed15b7ac5e8c8e38aaaacba01fbee65657be7808a90290f90b1b648 not found: ID does not exist"
Nov 03 21:27:04.713553 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.713529    5053 scope.go:122] "RemoveContainer" containerID="288fdff87ee272d32b860033ef32db18a1ff944e5830b97c229ae91d67455d9f"
Nov 03 21:27:04.713764 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:27:04.713747    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"288fdff87ee272d32b860033ef32db18a1ff944e5830b97c229ae91d67455d9f\": container with ID starting with 288fdff87ee272d32b860033ef32db18a1ff944e5830b97c229ae91d67455d9f not found: ID does not exist" containerID="288fdff87ee272d32b860033ef32db18a1ff944e5830b97c229ae91d67455d9f"
Nov 03 21:27:04.713803 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.713770    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"288fdff87ee272d32b860033ef32db18a1ff944e5830b97c229ae91d67455d9f"} err="failed to get container status \"288fdff87ee272d32b860033ef32db18a1ff944e5830b97c229ae91d67455d9f\": rpc error: code = NotFound desc = could not find container \"288fdff87ee272d32b860033ef32db18a1ff944e5830b97c229ae91d67455d9f\": container with ID starting with 288fdff87ee272d32b860033ef32db18a1ff944e5830b97c229ae91d67455d9f not found: ID does not exist"
Nov 03 21:27:04.857396 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.857368    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"]
Nov 03 21:27:04.861894 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:04.861868    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-35257bc0-openshift-default-7ff7595c99-5l9ph"]
Nov 03 21:27:05.667337 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:05.667305    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="610101d0-cf94-41b6-a21e-18e652eb9b03" path="/var/lib/kubelet/pods/610101d0-cf94-41b6-a21e-18e652eb9b03/volumes"
Nov 03 21:27:09.666499 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:09.666469    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt"
Nov 03 21:27:09.666910 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:09.666532    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h"
Nov 03 21:27:10.954885 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:10.954844    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9"]
Nov 03 21:27:10.955552 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:10.955532    5053 cpu_manager.go:414] "containerMap: removing container" podUID="610101d0-cf94-41b6-a21e-18e652eb9b03" containerName="istio-proxy"
Nov 03 21:27:10.955642 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:10.955553    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="610101d0-cf94-41b6-a21e-18e652eb9b03" containerName="istio-proxy"
Nov 03 21:27:10.955642 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:10.955573    5053 cpu_manager.go:414] "containerMap: removing container" podUID="610101d0-cf94-41b6-a21e-18e652eb9b03" containerName="istio-proxy"
Nov 03 21:27:10.955642 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:10.955581    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="610101d0-cf94-41b6-a21e-18e652eb9b03" containerName="istio-proxy"
Nov 03 21:27:10.955783 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:10.955721    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="610101d0-cf94-41b6-a21e-18e652eb9b03" containerName="istio-proxy"
Nov 03 21:27:10.955783 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:10.955736    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="610101d0-cf94-41b6-a21e-18e652eb9b03" containerName="istio-proxy"
Nov 03 21:27:10.960099 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:27:10.960072    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:27:10.960414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:10.960397    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9"
Nov 03 21:27:11.030199 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:11.030154    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9"]
Nov 03 21:27:11.039390 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:11.039361    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5s85\" (UniqueName: \"kubernetes.io/projected/4cee4995-fb03-43bc-a63d-f764dba58a39-kube-api-access-q5s85\") pod \"payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9\" (UID: \"4cee4995-fb03-43bc-a63d-f764dba58a39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9"
Nov 03 21:27:11.039579 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:11.039527    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cee4995-fb03-43bc-a63d-f764dba58a39-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9\" (UID: \"4cee4995-fb03-43bc-a63d-f764dba58a39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9"
Nov 03 21:27:11.062131 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:11.062095    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-6de8e6-dockercfg-cgrlf\""
Nov 03 21:27:11.062317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:11.062166    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-6de8e6\""
Nov 03 21:27:11.124262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:11.103979    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b"]
Nov 03 21:27:11.145346 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:11.145308    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5s85\" (UniqueName: \"kubernetes.io/projected/4cee4995-fb03-43bc-a63d-f764dba58a39-kube-api-access-q5s85\") pod \"payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9\" (UID: \"4cee4995-fb03-43bc-a63d-f764dba58a39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9"
Nov 03 21:27:11.145718 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:11.145689    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cee4995-fb03-43bc-a63d-f764dba58a39-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9\" (UID: \"4cee4995-fb03-43bc-a63d-f764dba58a39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9"
Nov 03 21:27:11.150287 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:11.146919    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cee4995-fb03-43bc-a63d-f764dba58a39-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9\" (UID: \"4cee4995-fb03-43bc-a63d-f764dba58a39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9"
Nov 03 21:27:11.156081 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:11.155636    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b"
Nov 03 21:27:11.163274 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:11.163221    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b"]
Nov 03 21:27:11.250792 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:11.250708    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwq5z\" (UniqueName: \"kubernetes.io/projected/b4036e26-b7be-4c14-bd68-4863da3c3fe0-kube-api-access-zwq5z\") pod \"payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b\" (UID: \"b4036e26-b7be-4c14-bd68-4863da3c3fe0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b"
Nov 03 21:27:11.251075 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:11.251050    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4036e26-b7be-4c14-bd68-4863da3c3fe0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b\" (UID: \"b4036e26-b7be-4c14-bd68-4863da3c3fe0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b"
Nov 03 21:27:11.254806 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:11.254752    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5s85\" (UniqueName: \"kubernetes.io/projected/4cee4995-fb03-43bc-a63d-f764dba58a39-kube-api-access-q5s85\") pod \"payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9\" (UID: \"4cee4995-fb03-43bc-a63d-f764dba58a39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9"
Nov 03 21:27:11.273403 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:11.273375    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9"
Nov 03 21:27:11.360110 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:11.355580    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwq5z\" (UniqueName: \"kubernetes.io/projected/b4036e26-b7be-4c14-bd68-4863da3c3fe0-kube-api-access-zwq5z\") pod \"payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b\" (UID: \"b4036e26-b7be-4c14-bd68-4863da3c3fe0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b"
Nov 03 21:27:11.360110 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:11.355675    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4036e26-b7be-4c14-bd68-4863da3c3fe0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b\" (UID: \"b4036e26-b7be-4c14-bd68-4863da3c3fe0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b"
Nov 03 21:27:11.360110 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:11.356760    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4036e26-b7be-4c14-bd68-4863da3c3fe0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b\" (UID: \"b4036e26-b7be-4c14-bd68-4863da3c3fe0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b"
Nov 03 21:27:11.427550 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:11.427487    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwq5z\" (UniqueName: \"kubernetes.io/projected/b4036e26-b7be-4c14-bd68-4863da3c3fe0-kube-api-access-zwq5z\") pod \"payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b\" (UID: \"b4036e26-b7be-4c14-bd68-4863da3c3fe0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b"
Nov 03 21:27:11.482396 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:11.482362    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b"
Nov 03 21:27:11.668761 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:11.668733    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9"]
Nov 03 21:27:11.670223 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:27:11.670194    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cee4995_fb03_43bc_a63d_f764dba58a39.slice/crio-5aaff74c73fe33b896c2521e6d0ee3f780693d726a88829afda134d38c21908c WatchSource:0}: Error finding container 5aaff74c73fe33b896c2521e6d0ee3f780693d726a88829afda134d38c21908c: Status 404 returned error can't find the container with id 5aaff74c73fe33b896c2521e6d0ee3f780693d726a88829afda134d38c21908c
Nov 03 21:27:11.746056 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:11.746017    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9" event={"ID":"4cee4995-fb03-43bc-a63d-f764dba58a39","Type":"ContainerStarted","Data":"5aaff74c73fe33b896c2521e6d0ee3f780693d726a88829afda134d38c21908c"}
Nov 03 21:27:11.888191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:11.888166    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b"]
Nov 03 21:27:11.890160 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:27:11.890133    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4036e26_b7be_4c14_bd68_4863da3c3fe0.slice/crio-0e7f493c234b1074fbbc6cf0aa4dd365b825c35830f1502050d5e685553017bf WatchSource:0}: Error finding container 0e7f493c234b1074fbbc6cf0aa4dd365b825c35830f1502050d5e685553017bf: Status 404 returned error can't find the container with id 0e7f493c234b1074fbbc6cf0aa4dd365b825c35830f1502050d5e685553017bf
Nov 03 21:27:12.753460 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:12.753421    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b" event={"ID":"b4036e26-b7be-4c14-bd68-4863da3c3fe0","Type":"ContainerStarted","Data":"dff7de5b02b5d5a738a6e8e311a95694d4761c80d6a740c6a35354ac743898b2"}
Nov 03 21:27:12.753952 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:12.753484    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b" event={"ID":"b4036e26-b7be-4c14-bd68-4863da3c3fe0","Type":"ContainerStarted","Data":"0e7f493c234b1074fbbc6cf0aa4dd365b825c35830f1502050d5e685553017bf"}
Nov 03 21:27:12.753952 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:12.753513    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b"
Nov 03 21:27:12.755082 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:12.755056    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9" event={"ID":"4cee4995-fb03-43bc-a63d-f764dba58a39","Type":"ContainerStarted","Data":"cd8dc383b439f79e13e223c48e54410069c13ff033541735d99b9980018e9fda"}
Nov 03 21:27:12.755209 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:12.755100    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9"
Nov 03 21:27:12.878788 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:12.878734    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b" podStartSLOduration=1.878718074 podStartE2EDuration="1.878718074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:12.876337448 +0000 UTC m=+747.370259322" watchObservedRunningTime="2026-11-03 21:27:12.878718074 +0000 UTC m=+747.372639957"
Nov 03 21:27:12.988862 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:12.988735    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9" podStartSLOduration=2.988715195 podStartE2EDuration="2.988715195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:12.988068016 +0000 UTC m=+747.481989892" watchObservedRunningTime="2026-11-03 21:27:12.988715195 +0000 UTC m=+747.482637069"
Nov 03 21:27:17.126306 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:17.126268    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8"]
Nov 03 21:27:17.176690 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:17.176653    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-daf21c29-jwxm8"]
Nov 03 21:27:17.665735 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:17.665703    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19e966b7-8a93-48ed-85eb-91dc76651258" path="/var/lib/kubelet/pods/19e966b7-8a93-48ed-85eb-91dc76651258/volumes"
Nov 03 21:27:18.796417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:18.796381    5053 generic.go:356] "Generic (PLEG): container finished" podID="88e0381b-b02d-40f1-acef-babc6c0248d1" containerID="6c9b7abbf29d54e93d7391907f9f88d1229fa2e85431be99a15107ad56df6380" exitCode=0
Nov 03 21:27:18.796876 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:18.796451    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj" event={"ID":"88e0381b-b02d-40f1-acef-babc6c0248d1","Type":"ContainerDied","Data":"6c9b7abbf29d54e93d7391907f9f88d1229fa2e85431be99a15107ad56df6380"}
Nov 03 21:27:18.796876 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:18.796823    5053 scope.go:122] "RemoveContainer" containerID="6c9b7abbf29d54e93d7391907f9f88d1229fa2e85431be99a15107ad56df6380"
Nov 03 21:27:18.798160 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:18.798132    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:27:18.798226 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:18.798210    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:27:18.798291 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:18.798265    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:27:18.838262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:18.838210    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"]
Nov 03 21:27:18.843478 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:18.843458    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:18.897919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:18.897891    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-f463c9-openshift-default-dockercfg-jbvbq\""
Nov 03 21:27:18.938611 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:18.938575    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"]
Nov 03 21:27:18.973841 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:18.973808    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-data\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:18.974017 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:18.973919    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-token\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:18.974017 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:18.973961    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btnkn\" (UniqueName: \"kubernetes.io/projected/f9207d7a-c808-4e41-8e12-c3219c4f8215-kube-api-access-btnkn\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:18.974017 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:18.974003    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-workload-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:18.974121 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:18.974082    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-podinfo\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:18.974156 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:18.974120    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-envoy\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:18.974188 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:18.974153    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9207d7a-c808-4e41-8e12-c3219c4f8215-istiod-ca-cert\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:18.974220 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:18.974186    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-workload-certs\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:18.974332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:18.974276    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-credential-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:19.074943 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.074846    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-data\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:19.074943 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.074902    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-token\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:19.074943 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.074929    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btnkn\" (UniqueName: \"kubernetes.io/projected/f9207d7a-c808-4e41-8e12-c3219c4f8215-kube-api-access-btnkn\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:19.075228 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.074964    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-workload-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:19.075228 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.075032    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-podinfo\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:19.075228 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.075067    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-envoy\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:19.075228 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.075101    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9207d7a-c808-4e41-8e12-c3219c4f8215-istiod-ca-cert\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:19.075228 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.075131    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-workload-certs\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:19.075228 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.075198    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-credential-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:19.075533 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.075282    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-data\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:19.075533 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.075416    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-workload-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:19.075633 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.075570    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-workload-certs\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:19.075672 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.075641    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-credential-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:19.076074 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.076050    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9207d7a-c808-4e41-8e12-c3219c4f8215-istiod-ca-cert\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:19.077626 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.077604    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-envoy\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:19.077911 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.077891    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-podinfo\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:19.136934 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.134732    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-token\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:19.141283 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.138925    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btnkn\" (UniqueName: \"kubernetes.io/projected/f9207d7a-c808-4e41-8e12-c3219c4f8215-kube-api-access-btnkn\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:19.175175 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.175129    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:19.637845 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.637802    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"]
Nov 03 21:27:19.663987 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:27:19.663951    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9207d7a_c808_4e41_8e12_c3219c4f8215.slice/crio-2a9b16e464138fc47c757a7618d3e24a98c56f8a6d0e7d9a78d2a43776761b82 WatchSource:0}: Error finding container 2a9b16e464138fc47c757a7618d3e24a98c56f8a6d0e7d9a78d2a43776761b82: Status 404 returned error can't find the container with id 2a9b16e464138fc47c757a7618d3e24a98c56f8a6d0e7d9a78d2a43776761b82
Nov 03 21:27:19.666274 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.665907    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:27:19.666274 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.665999    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:27:19.666274 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.666044    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:27:19.822161 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.821171    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z" event={"ID":"f9207d7a-c808-4e41-8e12-c3219c4f8215","Type":"ContainerStarted","Data":"411eab61a7165a43acc3fb20510a57505c42e0aea66d11c93c5106ba4cb37d1f"}
Nov 03 21:27:19.822161 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.821259    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z" event={"ID":"f9207d7a-c808-4e41-8e12-c3219c4f8215","Type":"ContainerStarted","Data":"2a9b16e464138fc47c757a7618d3e24a98c56f8a6d0e7d9a78d2a43776761b82"}
Nov 03 21:27:19.833627 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:19.832567    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj" event={"ID":"88e0381b-b02d-40f1-acef-babc6c0248d1","Type":"ContainerStarted","Data":"2b600b493165ff5ed7b542af6da27f090da32cbcbd1c0605b93ccec390622bc8"}
Nov 03 21:27:20.086278 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:20.085491    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z" podStartSLOduration=2.085477789 podStartE2EDuration="2.085477789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:19.963577769 +0000 UTC m=+754.457499639" watchObservedRunningTime="2026-11-03 21:27:20.085477789 +0000 UTC m=+754.579399663"
Nov 03 21:27:20.176199 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:20.176166    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:20.177146 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:20.177116    5053 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 21:27:20.177352 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:20.177167    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z" podUID="f9207d7a-c808-4e41-8e12-c3219c4f8215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 21:27:20.543257 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:20.542704    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:27:20.543257 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:20.542839    5053 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 03 21:27:20.543257 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:20.542886    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj" podUID="88e0381b-b02d-40f1-acef-babc6c0248d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 03 21:27:21.193139 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:21.193104    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:21.543321 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:21.542657    5053 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 03 21:27:21.543321 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:21.542720    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj" podUID="88e0381b-b02d-40f1-acef-babc6c0248d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 03 21:27:21.866557 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:21.866478    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:21.867648 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:21.867632    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:27:22.542700 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:22.542660    5053 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 03 21:27:22.543323 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:22.543288    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj" podUID="88e0381b-b02d-40f1-acef-babc6c0248d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 03 21:27:23.543676 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:23.543639    5053 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 03 21:27:23.544321 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:23.544288    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj" podUID="88e0381b-b02d-40f1-acef-babc6c0248d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 03 21:27:23.765768 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:23.765744    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b"
Nov 03 21:27:23.766169 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:23.766154    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9"
Nov 03 21:27:24.548262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:24.547731    5053 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 03 21:27:24.548262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:24.547784    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj" podUID="88e0381b-b02d-40f1-acef-babc6c0248d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 03 21:27:25.547415 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:25.547332    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:27:25.547736 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:25.547716    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:27:25.548503 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:25.548480    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 21:27:28.040468 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.040428    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46"]
Nov 03 21:27:28.048460 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.048435    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46"
Nov 03 21:27:28.101012 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.100978    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46"]
Nov 03 21:27:28.125612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.125362    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjdl7\" (UniqueName: \"kubernetes.io/projected/59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d-kube-api-access-xjdl7\") pod \"payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46\" (UID: \"59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46"
Nov 03 21:27:28.125612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.125486    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46\" (UID: \"59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46"
Nov 03 21:27:28.148417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.147810    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-f463c9-dockercfg-s84fj\""
Nov 03 21:27:28.148417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.147964    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-f463c9\""
Nov 03 21:27:28.192126 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.192089    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d"]
Nov 03 21:27:28.195913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.195889    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d"
Nov 03 21:27:28.226893 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.226866    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46\" (UID: \"59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46"
Nov 03 21:27:28.227032 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.226975    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjdl7\" (UniqueName: \"kubernetes.io/projected/59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d-kube-api-access-xjdl7\") pod \"payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46\" (UID: \"59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46"
Nov 03 21:27:28.227582 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.227558    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46\" (UID: \"59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46"
Nov 03 21:27:28.252000 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.251971    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d"]
Nov 03 21:27:28.283573 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.283544    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjdl7\" (UniqueName: \"kubernetes.io/projected/59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d-kube-api-access-xjdl7\") pod \"payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46\" (UID: \"59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46"
Nov 03 21:27:28.328302 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.328212    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpkdg\" (UniqueName: \"kubernetes.io/projected/463cdfee-c0ab-4bc1-b51f-20da775292f9-kube-api-access-zpkdg\") pod \"payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d\" (UID: \"463cdfee-c0ab-4bc1-b51f-20da775292f9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d"
Nov 03 21:27:28.328302 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.328277    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/463cdfee-c0ab-4bc1-b51f-20da775292f9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d\" (UID: \"463cdfee-c0ab-4bc1-b51f-20da775292f9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d"
Nov 03 21:27:28.361720 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.361691    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46"
Nov 03 21:27:28.431131 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.429182    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpkdg\" (UniqueName: \"kubernetes.io/projected/463cdfee-c0ab-4bc1-b51f-20da775292f9-kube-api-access-zpkdg\") pod \"payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d\" (UID: \"463cdfee-c0ab-4bc1-b51f-20da775292f9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d"
Nov 03 21:27:28.431131 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.429223    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/463cdfee-c0ab-4bc1-b51f-20da775292f9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d\" (UID: \"463cdfee-c0ab-4bc1-b51f-20da775292f9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d"
Nov 03 21:27:28.431131 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.430149    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/463cdfee-c0ab-4bc1-b51f-20da775292f9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d\" (UID: \"463cdfee-c0ab-4bc1-b51f-20da775292f9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d"
Nov 03 21:27:28.485380 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.485354    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpkdg\" (UniqueName: \"kubernetes.io/projected/463cdfee-c0ab-4bc1-b51f-20da775292f9-kube-api-access-zpkdg\") pod \"payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d\" (UID: \"463cdfee-c0ab-4bc1-b51f-20da775292f9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d"
Nov 03 21:27:28.515061 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.515030    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d"
Nov 03 21:27:28.731386 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.731354    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46"]
Nov 03 21:27:28.733256 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:27:28.733215    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f5490d_1a63_49f9_8c8f_3d9a3c4f5c9d.slice/crio-75fa67c71757fbe60560b5a7e09913f878c21b87f3eecf48e5d81b1387dc38cf WatchSource:0}: Error finding container 75fa67c71757fbe60560b5a7e09913f878c21b87f3eecf48e5d81b1387dc38cf: Status 404 returned error can't find the container with id 75fa67c71757fbe60560b5a7e09913f878c21b87f3eecf48e5d81b1387dc38cf
Nov 03 21:27:28.897497 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.897473    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d"]
Nov 03 21:27:28.899011 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:27:28.898984    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463cdfee_c0ab_4bc1_b51f_20da775292f9.slice/crio-62c8286361151bae7fc3b2c683a352713827e0aa31d77894a3cef64756a221eb WatchSource:0}: Error finding container 62c8286361151bae7fc3b2c683a352713827e0aa31d77894a3cef64756a221eb: Status 404 returned error can't find the container with id 62c8286361151bae7fc3b2c683a352713827e0aa31d77894a3cef64756a221eb
Nov 03 21:27:28.927437 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.927412    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46" event={"ID":"59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d","Type":"ContainerStarted","Data":"6f403b91188771124d71b9c08ec2ed336f0efe383c824e21c94d6c6d533635e4"}
Nov 03 21:27:28.927551 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.927485    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46" event={"ID":"59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d","Type":"ContainerStarted","Data":"75fa67c71757fbe60560b5a7e09913f878c21b87f3eecf48e5d81b1387dc38cf"}
Nov 03 21:27:28.927551 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.927522    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46"
Nov 03 21:27:28.928854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.928831    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d" event={"ID":"463cdfee-c0ab-4bc1-b51f-20da775292f9","Type":"ContainerStarted","Data":"62c8286361151bae7fc3b2c683a352713827e0aa31d77894a3cef64756a221eb"}
Nov 03 21:27:28.930204 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.930183    5053 generic.go:356] "Generic (PLEG): container finished" podID="0a1fdc06-686e-499e-9c23-ae901773d3b6" containerID="83a33949ac671af1b3a7c54eaac732d1b65b39f42b70dd1145e73e94809fdd63" exitCode=0
Nov 03 21:27:28.930291 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.930275    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv" event={"ID":"0a1fdc06-686e-499e-9c23-ae901773d3b6","Type":"ContainerDied","Data":"83a33949ac671af1b3a7c54eaac732d1b65b39f42b70dd1145e73e94809fdd63"}
Nov 03 21:27:28.930590 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.930577    5053 scope.go:122] "RemoveContainer" containerID="83a33949ac671af1b3a7c54eaac732d1b65b39f42b70dd1145e73e94809fdd63"
Nov 03 21:27:28.931705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.931679    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:27:28.931766 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.931738    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:27:28.931800 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:28.931768    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:27:29.052181 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:29.052062    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46" podStartSLOduration=1.052045472 podStartE2EDuration="1.052045472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:29.051070828 +0000 UTC m=+763.544992703" watchObservedRunningTime="2026-11-03 21:27:29.052045472 +0000 UTC m=+763.545967348"
Nov 03 21:27:29.938045 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:29.937999    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d" event={"ID":"463cdfee-c0ab-4bc1-b51f-20da775292f9","Type":"ContainerStarted","Data":"33c0875f1def25cdd403bb84f56bc77ee254f6d4d74f2d2fcf91223703826795"}
Nov 03 21:27:29.938370 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:29.938071    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d"
Nov 03 21:27:29.940203 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:29.940178    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv" event={"ID":"0a1fdc06-686e-499e-9c23-ae901773d3b6","Type":"ContainerStarted","Data":"89da477e9c050923dbf5bcce3b3a9e3e812b78311979ac0a38bd896a6ad4e15b"}
Nov 03 21:27:30.057399 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:30.057344    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d" podStartSLOduration=2.057330089 podStartE2EDuration="2.057330089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:30.054541423 +0000 UTC m=+764.548463296" watchObservedRunningTime="2026-11-03 21:27:30.057330089 +0000 UTC m=+764.551251967"
Nov 03 21:27:30.260738 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:30.260701    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:27:30.261872 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:30.261845    5053 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 03 21:27:30.262009 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:30.261894    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv" podUID="0a1fdc06-686e-499e-9c23-ae901773d3b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 03 21:27:31.260528 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:31.260497    5053 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 03 21:27:31.260936 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:31.260560    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv" podUID="0a1fdc06-686e-499e-9c23-ae901773d3b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 03 21:27:32.261074 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:32.261034    5053 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 03 21:27:32.261534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:32.261088    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv" podUID="0a1fdc06-686e-499e-9c23-ae901773d3b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 03 21:27:33.260403 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:33.260369    5053 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 03 21:27:33.260588 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:33.260430    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv" podUID="0a1fdc06-686e-499e-9c23-ae901773d3b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 03 21:27:33.574208 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:33.574121    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:27:34.260865 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:34.260830    5053 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 03 21:27:34.261061 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:34.260889    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv" podUID="0a1fdc06-686e-499e-9c23-ae901773d3b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 03 21:27:35.289353 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:35.288411    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:27:35.290585 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:35.290563    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 21:27:39.943985 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:39.943957    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46"
Nov 03 21:27:40.950430 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:27:40.950392    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:27:40.950803 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:40.950610    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d"
Nov 03 21:27:45.059030 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:45.059000    5053 generic.go:356] "Generic (PLEG): container finished" podID="51d17433-dc86-491b-9276-080c84dfdc8e" containerID="77cc02f8698ec91ff89cc2168890c6970d6b9d597d41712c8ef537f3644d6f3f" exitCode=0
Nov 03 21:27:45.059399 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:45.059077    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd" event={"ID":"51d17433-dc86-491b-9276-080c84dfdc8e","Type":"ContainerDied","Data":"77cc02f8698ec91ff89cc2168890c6970d6b9d597d41712c8ef537f3644d6f3f"}
Nov 03 21:27:45.059570 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:45.059554    5053 scope.go:122] "RemoveContainer" containerID="77cc02f8698ec91ff89cc2168890c6970d6b9d597d41712c8ef537f3644d6f3f"
Nov 03 21:27:45.060846 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:45.060822    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:27:45.060899 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:45.060885    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:27:45.060932 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:45.060917    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:27:46.066775 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:46.066690    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd" event={"ID":"51d17433-dc86-491b-9276-080c84dfdc8e","Type":"ContainerStarted","Data":"b2d879b06a72cda2dd3d60517e935ff43844f9713ce2f1249ab29dcb6331a45b"}
Nov 03 21:27:46.824742 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:46.824709    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:46.825146 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:46.825119    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 21:27:46.825358 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:46.825332    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd" podUID="51d17433-dc86-491b-9276-080c84dfdc8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 21:27:47.821270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:47.813690    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 21:27:47.821270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:47.813751    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd" podUID="51d17433-dc86-491b-9276-080c84dfdc8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 21:27:48.686875 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:48.686842    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:48.810265 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:48.810177    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 21:27:48.810858 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:48.810470    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd" podUID="51d17433-dc86-491b-9276-080c84dfdc8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 21:27:49.819263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:49.813658    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 21:27:49.819263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:49.813731    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd" podUID="51d17433-dc86-491b-9276-080c84dfdc8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 21:27:50.810555 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:50.810511    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 21:27:50.810740 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:50.810595    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd" podUID="51d17433-dc86-491b-9276-080c84dfdc8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 21:27:51.814302 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:51.814270    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:27:51.815344 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:27:51.815324    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:28:10.926974 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:28:10.926877    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:28:11.257057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:11.257027    5053 generic.go:356] "Generic (PLEG): container finished" podID="f9207d7a-c808-4e41-8e12-c3219c4f8215" containerID="411eab61a7165a43acc3fb20510a57505c42e0aea66d11c93c5106ba4cb37d1f" exitCode=0
Nov 03 21:28:11.257228 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:11.257100    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z" event={"ID":"f9207d7a-c808-4e41-8e12-c3219c4f8215","Type":"ContainerDied","Data":"411eab61a7165a43acc3fb20510a57505c42e0aea66d11c93c5106ba4cb37d1f"}
Nov 03 21:28:11.257547 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:11.257529    5053 scope.go:122] "RemoveContainer" containerID="411eab61a7165a43acc3fb20510a57505c42e0aea66d11c93c5106ba4cb37d1f"
Nov 03 21:28:11.258920 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:11.258887    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:28:11.259000 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:11.258960    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:28:11.259000 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:11.258990    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:28:12.266066 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:12.266032    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z" event={"ID":"f9207d7a-c808-4e41-8e12-c3219c4f8215","Type":"ContainerStarted","Data":"17504f2718462eca7940beed493d3a0aaf7bb6c65be1abb1bfb081894d51fa00"}
Nov 03 21:28:13.175337 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:13.175298    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:28:13.175541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:13.175520    5053 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 21:28:13.175588 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:13.175570    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z" podUID="f9207d7a-c808-4e41-8e12-c3219c4f8215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 21:28:14.175775 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:14.175735    5053 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 21:28:14.176143 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:14.175800    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z" podUID="f9207d7a-c808-4e41-8e12-c3219c4f8215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 21:28:15.176483 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:15.176449    5053 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 21:28:15.176873 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:15.176499    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z" podUID="f9207d7a-c808-4e41-8e12-c3219c4f8215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 21:28:16.175643 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:16.175604    5053 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 21:28:16.175819 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:16.175663    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z" podUID="f9207d7a-c808-4e41-8e12-c3219c4f8215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 21:28:17.196716 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:17.196677    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:28:17.197189 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:17.197107    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:28:17.197902 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:17.197886    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 21:28:31.034595 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.034563    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"]
Nov 03 21:28:31.037486 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.037466    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:28:31.095073 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.095028    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"]
Nov 03 21:28:31.135032 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.135001    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/512b3ab1-45ba-4f96-9799-b129b25fca42-tls-certs\") pod \"simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n\" (UID: \"512b3ab1-45ba-4f96-9799-b129b25fca42\") " pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:28:31.135215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.135059    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-tmp-dir\") pod \"simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n\" (UID: \"512b3ab1-45ba-4f96-9799-b129b25fca42\") " pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:28:31.135215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.135077    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25h56\" (UniqueName: \"kubernetes.io/projected/512b3ab1-45ba-4f96-9799-b129b25fca42-kube-api-access-25h56\") pod \"simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n\" (UID: \"512b3ab1-45ba-4f96-9799-b129b25fca42\") " pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:28:31.135330 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.135272    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-home\") pod \"simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n\" (UID: \"512b3ab1-45ba-4f96-9799-b129b25fca42\") " pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:28:31.135371 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.135335    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-model-cache\") pod \"simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n\" (UID: \"512b3ab1-45ba-4f96-9799-b129b25fca42\") " pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:28:31.135414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.135383    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-dshm\") pod \"simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n\" (UID: \"512b3ab1-45ba-4f96-9799-b129b25fca42\") " pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:28:31.141265 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.141223    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6de8e6\"/\"kube-root-ca.crt\""
Nov 03 21:28:31.141417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.141365    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6de8e6\"/\"openshift-service-ca.crt\""
Nov 03 21:28:31.141482 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.141419    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6de8e6\"/\"simulator-w4-6de8e6-kserve-self-signed-certs\""
Nov 03 21:28:31.141482 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.141437    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6de8e6\"/\"default-dockercfg-mfx2t\""
Nov 03 21:28:31.236034 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.236001    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-model-cache\") pod \"simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n\" (UID: \"512b3ab1-45ba-4f96-9799-b129b25fca42\") " pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:28:31.236214 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.236046    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-dshm\") pod \"simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n\" (UID: \"512b3ab1-45ba-4f96-9799-b129b25fca42\") " pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:28:31.236214 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.236103    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/512b3ab1-45ba-4f96-9799-b129b25fca42-tls-certs\") pod \"simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n\" (UID: \"512b3ab1-45ba-4f96-9799-b129b25fca42\") " pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:28:31.236214 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.236157    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-tmp-dir\") pod \"simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n\" (UID: \"512b3ab1-45ba-4f96-9799-b129b25fca42\") " pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:28:31.236214 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.236178    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25h56\" (UniqueName: \"kubernetes.io/projected/512b3ab1-45ba-4f96-9799-b129b25fca42-kube-api-access-25h56\") pod \"simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n\" (UID: \"512b3ab1-45ba-4f96-9799-b129b25fca42\") " pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:28:31.236527 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.236248    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-home\") pod \"simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n\" (UID: \"512b3ab1-45ba-4f96-9799-b129b25fca42\") " pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:28:31.236527 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.236489    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-model-cache\") pod \"simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n\" (UID: \"512b3ab1-45ba-4f96-9799-b129b25fca42\") " pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:28:31.236642 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.236557    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-tmp-dir\") pod \"simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n\" (UID: \"512b3ab1-45ba-4f96-9799-b129b25fca42\") " pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:28:31.236642 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.236625    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-home\") pod \"simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n\" (UID: \"512b3ab1-45ba-4f96-9799-b129b25fca42\") " pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:28:31.238515 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.238491    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-dshm\") pod \"simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n\" (UID: \"512b3ab1-45ba-4f96-9799-b129b25fca42\") " pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:28:31.238787 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.238772    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/512b3ab1-45ba-4f96-9799-b129b25fca42-tls-certs\") pod \"simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n\" (UID: \"512b3ab1-45ba-4f96-9799-b129b25fca42\") " pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:28:31.293811 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.293747    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25h56\" (UniqueName: \"kubernetes.io/projected/512b3ab1-45ba-4f96-9799-b129b25fca42-kube-api-access-25h56\") pod \"simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n\" (UID: \"512b3ab1-45ba-4f96-9799-b129b25fca42\") " pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:28:31.355509 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.355470    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:28:31.729831 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:31.729795    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"]
Nov 03 21:28:31.731778 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:28:31.731755    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod512b3ab1_45ba_4f96_9799_b129b25fca42.slice/crio-fff1e89092d2d489c3c33c26190dda886bd63fbef34c04c6bc80560732500556 WatchSource:0}: Error finding container fff1e89092d2d489c3c33c26190dda886bd63fbef34c04c6bc80560732500556: Status 404 returned error can't find the container with id fff1e89092d2d489c3c33c26190dda886bd63fbef34c04c6bc80560732500556
Nov 03 21:28:32.384535 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:32.384500    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n" event={"ID":"512b3ab1-45ba-4f96-9799-b129b25fca42","Type":"ContainerStarted","Data":"ea8eff9b3ce3e0af6dc6411f546ca09e4d2d311835ef7b61935bab7cf41944d5"}
Nov 03 21:28:32.384535 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:32.384539    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n" event={"ID":"512b3ab1-45ba-4f96-9799-b129b25fca42","Type":"ContainerStarted","Data":"fff1e89092d2d489c3c33c26190dda886bd63fbef34c04c6bc80560732500556"}
Nov 03 21:28:32.498457 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:32.498403    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n" podStartSLOduration=2.498388153 podStartE2EDuration="2.498388153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:28:32.496795763 +0000 UTC m=+826.990717637" watchObservedRunningTime="2026-11-03 21:28:32.498388153 +0000 UTC m=+826.992310026"
Nov 03 21:28:33.356609 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:33.356571    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:28:40.927637 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:28:40.927597    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:28:41.355919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:41.355884    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:28:41.368119 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:41.368095    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:28:41.447220 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:28:41.447192    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:29:10.929275 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:29:10.929207    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:29:32.936772 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:32.936737    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"]
Nov 03 21:29:32.940701 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:32.940678    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:29:33.026995 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.026960    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-model-cache\") pod \"premium-w4-6de8e6-kserve-76bc9fd78-lbkxx\" (UID: \"a6c0d3aa-18c2-465b-846b-b82968acc780\") " pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:29:33.027183 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.027016    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgdf5\" (UniqueName: \"kubernetes.io/projected/a6c0d3aa-18c2-465b-846b-b82968acc780-kube-api-access-lgdf5\") pod \"premium-w4-6de8e6-kserve-76bc9fd78-lbkxx\" (UID: \"a6c0d3aa-18c2-465b-846b-b82968acc780\") " pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:29:33.027183 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.027097    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-tmp-dir\") pod \"premium-w4-6de8e6-kserve-76bc9fd78-lbkxx\" (UID: \"a6c0d3aa-18c2-465b-846b-b82968acc780\") " pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:29:33.027183 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.027168    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c0d3aa-18c2-465b-846b-b82968acc780-tls-certs\") pod \"premium-w4-6de8e6-kserve-76bc9fd78-lbkxx\" (UID: \"a6c0d3aa-18c2-465b-846b-b82968acc780\") " pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:29:33.027412 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.027205    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-dshm\") pod \"premium-w4-6de8e6-kserve-76bc9fd78-lbkxx\" (UID: \"a6c0d3aa-18c2-465b-846b-b82968acc780\") " pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:29:33.027412 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.027230    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-home\") pod \"premium-w4-6de8e6-kserve-76bc9fd78-lbkxx\" (UID: \"a6c0d3aa-18c2-465b-846b-b82968acc780\") " pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:29:33.042876 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.042846    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6de8e6\"/\"premium-w4-6de8e6-kserve-self-signed-certs\""
Nov 03 21:29:33.045486 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.045464    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"]
Nov 03 21:29:33.109672 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.109636    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa52af6-tmzrg"]
Nov 03 21:29:33.113510 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.113489    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa52af6-tmzrg"
Nov 03 21:29:33.127952 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.127921    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgdf5\" (UniqueName: \"kubernetes.io/projected/a6c0d3aa-18c2-465b-846b-b82968acc780-kube-api-access-lgdf5\") pod \"premium-w4-6de8e6-kserve-76bc9fd78-lbkxx\" (UID: \"a6c0d3aa-18c2-465b-846b-b82968acc780\") " pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:29:33.128115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.127969    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-tmp-dir\") pod \"premium-w4-6de8e6-kserve-76bc9fd78-lbkxx\" (UID: \"a6c0d3aa-18c2-465b-846b-b82968acc780\") " pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:29:33.128115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.128042    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c0d3aa-18c2-465b-846b-b82968acc780-tls-certs\") pod \"premium-w4-6de8e6-kserve-76bc9fd78-lbkxx\" (UID: \"a6c0d3aa-18c2-465b-846b-b82968acc780\") " pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:29:33.128115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.128067    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-dshm\") pod \"premium-w4-6de8e6-kserve-76bc9fd78-lbkxx\" (UID: \"a6c0d3aa-18c2-465b-846b-b82968acc780\") " pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:29:33.128115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.128097    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-home\") pod \"premium-w4-6de8e6-kserve-76bc9fd78-lbkxx\" (UID: \"a6c0d3aa-18c2-465b-846b-b82968acc780\") " pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:29:33.128346 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.128314    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-model-cache\") pod \"premium-w4-6de8e6-kserve-76bc9fd78-lbkxx\" (UID: \"a6c0d3aa-18c2-465b-846b-b82968acc780\") " pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:29:33.128423 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.128398    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-tmp-dir\") pod \"premium-w4-6de8e6-kserve-76bc9fd78-lbkxx\" (UID: \"a6c0d3aa-18c2-465b-846b-b82968acc780\") " pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:29:33.128616 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.128560    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-home\") pod \"premium-w4-6de8e6-kserve-76bc9fd78-lbkxx\" (UID: \"a6c0d3aa-18c2-465b-846b-b82968acc780\") " pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:29:33.128740 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.128711    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-model-cache\") pod \"premium-w4-6de8e6-kserve-76bc9fd78-lbkxx\" (UID: \"a6c0d3aa-18c2-465b-846b-b82968acc780\") " pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:29:33.130568 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.130545    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-dshm\") pod \"premium-w4-6de8e6-kserve-76bc9fd78-lbkxx\" (UID: \"a6c0d3aa-18c2-465b-846b-b82968acc780\") " pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:29:33.130832 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.130817    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c0d3aa-18c2-465b-846b-b82968acc780-tls-certs\") pod \"premium-w4-6de8e6-kserve-76bc9fd78-lbkxx\" (UID: \"a6c0d3aa-18c2-465b-846b-b82968acc780\") " pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:29:33.166219 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.166174    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:29:33.166435 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.166277    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:29:33.166435 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.166296    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:29:33.166435 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.166286    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:29:33.169750 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.169713    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa52af6-tmzrg"]
Nov 03 21:29:33.184544 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.184508    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgdf5\" (UniqueName: \"kubernetes.io/projected/a6c0d3aa-18c2-465b-846b-b82968acc780-kube-api-access-lgdf5\") pod \"premium-w4-6de8e6-kserve-76bc9fd78-lbkxx\" (UID: \"a6c0d3aa-18c2-465b-846b-b82968acc780\") " pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:29:33.229076 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.229045    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e9dc7bc-a061-4404-84c3-23b3949232b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaa52af6-tmzrg\" (UID: \"9e9dc7bc-a061-4404-84c3-23b3949232b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa52af6-tmzrg"
Nov 03 21:29:33.229292 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.229110    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hcqc\" (UniqueName: \"kubernetes.io/projected/9e9dc7bc-a061-4404-84c3-23b3949232b0-kube-api-access-2hcqc\") pod \"maas-api-revoke-sub-aaa52af6-tmzrg\" (UID: \"9e9dc7bc-a061-4404-84c3-23b3949232b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa52af6-tmzrg"
Nov 03 21:29:33.229292 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.229257    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e9dc7bc-a061-4404-84c3-23b3949232b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaa52af6-tmzrg\" (UID: \"9e9dc7bc-a061-4404-84c3-23b3949232b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa52af6-tmzrg"
Nov 03 21:29:33.253485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.253437    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:29:33.333560 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.333476    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e9dc7bc-a061-4404-84c3-23b3949232b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaa52af6-tmzrg\" (UID: \"9e9dc7bc-a061-4404-84c3-23b3949232b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa52af6-tmzrg"
Nov 03 21:29:33.333702 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.333579    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hcqc\" (UniqueName: \"kubernetes.io/projected/9e9dc7bc-a061-4404-84c3-23b3949232b0-kube-api-access-2hcqc\") pod \"maas-api-revoke-sub-aaa52af6-tmzrg\" (UID: \"9e9dc7bc-a061-4404-84c3-23b3949232b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa52af6-tmzrg"
Nov 03 21:29:33.333702 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.333650    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e9dc7bc-a061-4404-84c3-23b3949232b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaa52af6-tmzrg\" (UID: \"9e9dc7bc-a061-4404-84c3-23b3949232b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa52af6-tmzrg"
Nov 03 21:29:33.334483 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.334448    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e9dc7bc-a061-4404-84c3-23b3949232b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaa52af6-tmzrg\" (UID: \"9e9dc7bc-a061-4404-84c3-23b3949232b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa52af6-tmzrg"
Nov 03 21:29:33.339985 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.336426    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e9dc7bc-a061-4404-84c3-23b3949232b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaa52af6-tmzrg\" (UID: \"9e9dc7bc-a061-4404-84c3-23b3949232b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa52af6-tmzrg"
Nov 03 21:29:33.390951 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.390919    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hcqc\" (UniqueName: \"kubernetes.io/projected/9e9dc7bc-a061-4404-84c3-23b3949232b0-kube-api-access-2hcqc\") pod \"maas-api-revoke-sub-aaa52af6-tmzrg\" (UID: \"9e9dc7bc-a061-4404-84c3-23b3949232b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa52af6-tmzrg"
Nov 03 21:29:33.427179 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.427144    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa52af6-tmzrg"
Nov 03 21:29:33.649631 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.649605    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"]
Nov 03 21:29:33.650899 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:29:33.650871    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6c0d3aa_18c2_465b_846b_b82968acc780.slice/crio-8b16c1530c78751acfd6ce77276711475e0010363a56d42cb96f6c94819a2678 WatchSource:0}: Error finding container 8b16c1530c78751acfd6ce77276711475e0010363a56d42cb96f6c94819a2678: Status 404 returned error can't find the container with id 8b16c1530c78751acfd6ce77276711475e0010363a56d42cb96f6c94819a2678
Nov 03 21:29:33.767208 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.767128    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx" event={"ID":"a6c0d3aa-18c2-465b-846b-b82968acc780","Type":"ContainerStarted","Data":"d2f61c28ec45b9976a0984df736177a90e6da2f254aa92fcd5e0887a20f26136"}
Nov 03 21:29:33.767208 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.767166    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx" event={"ID":"a6c0d3aa-18c2-465b-846b-b82968acc780","Type":"ContainerStarted","Data":"8b16c1530c78751acfd6ce77276711475e0010363a56d42cb96f6c94819a2678"}
Nov 03 21:29:33.799312 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.799286    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa52af6-tmzrg"]
Nov 03 21:29:33.801038 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:29:33.801015    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e9dc7bc_a061_4404_84c3_23b3949232b0.slice/crio-5f4d2741c45390ee80e416a05b14443a0164b6a5ab1b69026b3e19c46319f618 WatchSource:0}: Error finding container 5f4d2741c45390ee80e416a05b14443a0164b6a5ab1b69026b3e19c46319f618: Status 404 returned error can't find the container with id 5f4d2741c45390ee80e416a05b14443a0164b6a5ab1b69026b3e19c46319f618
Nov 03 21:29:33.881231 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:33.881172    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx" podStartSLOduration=1.881157008 podStartE2EDuration="1.881157008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:33.880173206 +0000 UTC m=+888.374095081" watchObservedRunningTime="2026-11-03 21:29:33.881157008 +0000 UTC m=+888.375078894"
Nov 03 21:29:34.254033 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:34.253996    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:29:34.774549 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:34.774512    5053 generic.go:356] "Generic (PLEG): container finished" podID="9e9dc7bc-a061-4404-84c3-23b3949232b0" containerID="24fcbd5704f938123df1900914a29a4cfa879954cd9e6d2729c4dcbb60fae7d5" exitCode=0
Nov 03 21:29:34.774770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:34.774589    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa52af6-tmzrg" event={"ID":"9e9dc7bc-a061-4404-84c3-23b3949232b0","Type":"ContainerDied","Data":"24fcbd5704f938123df1900914a29a4cfa879954cd9e6d2729c4dcbb60fae7d5"}
Nov 03 21:29:34.774770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:34.774643    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa52af6-tmzrg" event={"ID":"9e9dc7bc-a061-4404-84c3-23b3949232b0","Type":"ContainerStarted","Data":"5f4d2741c45390ee80e416a05b14443a0164b6a5ab1b69026b3e19c46319f618"}
Nov 03 21:29:35.929330 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:35.929303    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa52af6-tmzrg"
Nov 03 21:29:36.065203 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:36.065122    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e9dc7bc-a061-4404-84c3-23b3949232b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e9dc7bc-a061-4404-84c3-23b3949232b0-maas-api-openshift-ca\") pod \"9e9dc7bc-a061-4404-84c3-23b3949232b0\" (UID: \"9e9dc7bc-a061-4404-84c3-23b3949232b0\") "
Nov 03 21:29:36.065496 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:36.065461    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e9dc7bc-a061-4404-84c3-23b3949232b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e9dc7bc-a061-4404-84c3-23b3949232b0-maas-api-xks-ca\") pod \"9e9dc7bc-a061-4404-84c3-23b3949232b0\" (UID: \"9e9dc7bc-a061-4404-84c3-23b3949232b0\") "
Nov 03 21:29:36.065605 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:36.065523    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e9dc7bc-a061-4404-84c3-23b3949232b0-kube-api-access-2hcqc\" (UniqueName: \"kubernetes.io/projected/9e9dc7bc-a061-4404-84c3-23b3949232b0-kube-api-access-2hcqc\") pod \"9e9dc7bc-a061-4404-84c3-23b3949232b0\" (UID: \"9e9dc7bc-a061-4404-84c3-23b3949232b0\") "
Nov 03 21:29:36.065605 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:36.065575    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e9dc7bc-a061-4404-84c3-23b3949232b0-maas-api-openshift-ca" pod "9e9dc7bc-a061-4404-84c3-23b3949232b0" (UID: "9e9dc7bc-a061-4404-84c3-23b3949232b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:29:36.066071 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:36.066044    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e9dc7bc-a061-4404-84c3-23b3949232b0-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:29:36.067759 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:36.067736    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e9dc7bc-a061-4404-84c3-23b3949232b0-maas-api-xks-ca" pod "9e9dc7bc-a061-4404-84c3-23b3949232b0" (UID: "9e9dc7bc-a061-4404-84c3-23b3949232b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:29:36.068064 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:36.068035    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e9dc7bc-a061-4404-84c3-23b3949232b0-kube-api-access-2hcqc" pod "9e9dc7bc-a061-4404-84c3-23b3949232b0" (UID: "9e9dc7bc-a061-4404-84c3-23b3949232b0"). InnerVolumeSpecName "kube-api-access-2hcqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:29:36.167470 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:36.167431    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e9dc7bc-a061-4404-84c3-23b3949232b0-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:29:36.167470 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:36.167460    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hcqc\" (UniqueName: \"kubernetes.io/projected/9e9dc7bc-a061-4404-84c3-23b3949232b0-kube-api-access-2hcqc\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:29:36.789164 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:36.789130    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa52af6-tmzrg" event={"ID":"9e9dc7bc-a061-4404-84c3-23b3949232b0","Type":"ContainerDied","Data":"5f4d2741c45390ee80e416a05b14443a0164b6a5ab1b69026b3e19c46319f618"}
Nov 03 21:29:36.789379 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:36.789177    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f4d2741c45390ee80e416a05b14443a0164b6a5ab1b69026b3e19c46319f618"
Nov 03 21:29:36.789379 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:36.789182    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa52af6-tmzrg"
Nov 03 21:29:40.928398 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:29:40.928354    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:29:43.253877 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:43.253834    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:29:43.267124 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:43.267096    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:29:43.845409 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:43.845375    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:29:45.781888 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:45.781851    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 21:29:45.784927 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:45.784891    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 21:29:45.791320 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:45.791294    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:29:45.803467 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:45.803441    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:29:50.986042 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:50.986007    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f5a00ef-kpnvh"]
Nov 03 21:29:50.986657 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:50.986641    5053 cpu_manager.go:414] "containerMap: removing container" podUID="9e9dc7bc-a061-4404-84c3-23b3949232b0" containerName="revoke-keys"
Nov 03 21:29:50.986723 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:50.986659    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e9dc7bc-a061-4404-84c3-23b3949232b0" containerName="revoke-keys"
Nov 03 21:29:50.986763 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:50.986742    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e9dc7bc-a061-4404-84c3-23b3949232b0" containerName="revoke-keys"
Nov 03 21:29:50.990421 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:50.990404    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5a00ef-kpnvh"
Nov 03 21:29:51.046189 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:51.046157    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f5a00ef-kpnvh"]
Nov 03 21:29:51.054602 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:51.054566    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvxgt\" (UniqueName: \"kubernetes.io/projected/309804de-c1da-49fc-96b3-53ba8be575fa-kube-api-access-gvxgt\") pod \"maas-api-revoke-sub-2f5a00ef-kpnvh\" (UID: \"309804de-c1da-49fc-96b3-53ba8be575fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5a00ef-kpnvh"
Nov 03 21:29:51.054790 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:51.054773    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/309804de-c1da-49fc-96b3-53ba8be575fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f5a00ef-kpnvh\" (UID: \"309804de-c1da-49fc-96b3-53ba8be575fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5a00ef-kpnvh"
Nov 03 21:29:51.054848 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:51.054827    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/309804de-c1da-49fc-96b3-53ba8be575fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f5a00ef-kpnvh\" (UID: \"309804de-c1da-49fc-96b3-53ba8be575fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5a00ef-kpnvh"
Nov 03 21:29:51.092370 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:51.092335    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:29:51.092370 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:51.092341    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:29:51.093490 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:51.093475    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:29:51.093656 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:51.093642    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:29:51.155591 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:51.155555    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/309804de-c1da-49fc-96b3-53ba8be575fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f5a00ef-kpnvh\" (UID: \"309804de-c1da-49fc-96b3-53ba8be575fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5a00ef-kpnvh"
Nov 03 21:29:51.155776 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:51.155622    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/309804de-c1da-49fc-96b3-53ba8be575fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f5a00ef-kpnvh\" (UID: \"309804de-c1da-49fc-96b3-53ba8be575fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5a00ef-kpnvh"
Nov 03 21:29:51.155776 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:51.155698    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvxgt\" (UniqueName: \"kubernetes.io/projected/309804de-c1da-49fc-96b3-53ba8be575fa-kube-api-access-gvxgt\") pod \"maas-api-revoke-sub-2f5a00ef-kpnvh\" (UID: \"309804de-c1da-49fc-96b3-53ba8be575fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5a00ef-kpnvh"
Nov 03 21:29:51.156172 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:51.156150    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/309804de-c1da-49fc-96b3-53ba8be575fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f5a00ef-kpnvh\" (UID: \"309804de-c1da-49fc-96b3-53ba8be575fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5a00ef-kpnvh"
Nov 03 21:29:51.158160 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:51.158135    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/309804de-c1da-49fc-96b3-53ba8be575fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f5a00ef-kpnvh\" (UID: \"309804de-c1da-49fc-96b3-53ba8be575fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5a00ef-kpnvh"
Nov 03 21:29:51.212758 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:51.212727    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvxgt\" (UniqueName: \"kubernetes.io/projected/309804de-c1da-49fc-96b3-53ba8be575fa-kube-api-access-gvxgt\") pod \"maas-api-revoke-sub-2f5a00ef-kpnvh\" (UID: \"309804de-c1da-49fc-96b3-53ba8be575fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5a00ef-kpnvh"
Nov 03 21:29:51.303813 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:51.303731    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5a00ef-kpnvh"
Nov 03 21:29:51.679079 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:51.679055    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f5a00ef-kpnvh"]
Nov 03 21:29:51.680504 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:29:51.680477    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309804de_c1da_49fc_96b3_53ba8be575fa.slice/crio-34d15073e4fb036a626585eb9e65a9226946d65ef1d3ab87b67fef6f1edad6eb WatchSource:0}: Error finding container 34d15073e4fb036a626585eb9e65a9226946d65ef1d3ab87b67fef6f1edad6eb: Status 404 returned error can't find the container with id 34d15073e4fb036a626585eb9e65a9226946d65ef1d3ab87b67fef6f1edad6eb
Nov 03 21:29:51.884040 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:51.884002    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5a00ef-kpnvh" event={"ID":"309804de-c1da-49fc-96b3-53ba8be575fa","Type":"ContainerStarted","Data":"34d15073e4fb036a626585eb9e65a9226946d65ef1d3ab87b67fef6f1edad6eb"}
Nov 03 21:29:52.892184 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:52.892099    5053 generic.go:356] "Generic (PLEG): container finished" podID="309804de-c1da-49fc-96b3-53ba8be575fa" containerID="bef82bb3e5ee4eb7a6b369f41cd030a0d503687e94e0b0c3f205fdd363e8a0a5" exitCode=0
Nov 03 21:29:52.892586 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:52.892185    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5a00ef-kpnvh" event={"ID":"309804de-c1da-49fc-96b3-53ba8be575fa","Type":"ContainerDied","Data":"bef82bb3e5ee4eb7a6b369f41cd030a0d503687e94e0b0c3f205fdd363e8a0a5"}
Nov 03 21:29:54.045883 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:54.045859    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5a00ef-kpnvh"
Nov 03 21:29:54.086388 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:54.086359    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/309804de-c1da-49fc-96b3-53ba8be575fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/309804de-c1da-49fc-96b3-53ba8be575fa-maas-api-openshift-ca\") pod \"309804de-c1da-49fc-96b3-53ba8be575fa\" (UID: \"309804de-c1da-49fc-96b3-53ba8be575fa\") "
Nov 03 21:29:54.086564 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:54.086439    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/309804de-c1da-49fc-96b3-53ba8be575fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/309804de-c1da-49fc-96b3-53ba8be575fa-maas-api-xks-ca\") pod \"309804de-c1da-49fc-96b3-53ba8be575fa\" (UID: \"309804de-c1da-49fc-96b3-53ba8be575fa\") "
Nov 03 21:29:54.086564 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:54.086556    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/309804de-c1da-49fc-96b3-53ba8be575fa-kube-api-access-gvxgt\" (UniqueName: \"kubernetes.io/projected/309804de-c1da-49fc-96b3-53ba8be575fa-kube-api-access-gvxgt\") pod \"309804de-c1da-49fc-96b3-53ba8be575fa\" (UID: \"309804de-c1da-49fc-96b3-53ba8be575fa\") "
Nov 03 21:29:54.086746 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:54.086722    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/309804de-c1da-49fc-96b3-53ba8be575fa-maas-api-openshift-ca" pod "309804de-c1da-49fc-96b3-53ba8be575fa" (UID: "309804de-c1da-49fc-96b3-53ba8be575fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:29:54.087000 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:54.086977    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/309804de-c1da-49fc-96b3-53ba8be575fa-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:29:54.088684 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:54.088652    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/309804de-c1da-49fc-96b3-53ba8be575fa-kube-api-access-gvxgt" pod "309804de-c1da-49fc-96b3-53ba8be575fa" (UID: "309804de-c1da-49fc-96b3-53ba8be575fa"). InnerVolumeSpecName "kube-api-access-gvxgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:29:54.088775 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:54.088697    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/309804de-c1da-49fc-96b3-53ba8be575fa-maas-api-xks-ca" pod "309804de-c1da-49fc-96b3-53ba8be575fa" (UID: "309804de-c1da-49fc-96b3-53ba8be575fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:29:54.188653 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:54.188561    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvxgt\" (UniqueName: \"kubernetes.io/projected/309804de-c1da-49fc-96b3-53ba8be575fa-kube-api-access-gvxgt\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:29:54.188653 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:54.188588    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/309804de-c1da-49fc-96b3-53ba8be575fa-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:29:54.905753 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:54.905723    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5a00ef-kpnvh"
Nov 03 21:29:54.905937 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:54.905711    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5a00ef-kpnvh" event={"ID":"309804de-c1da-49fc-96b3-53ba8be575fa","Type":"ContainerDied","Data":"34d15073e4fb036a626585eb9e65a9226946d65ef1d3ab87b67fef6f1edad6eb"}
Nov 03 21:29:54.905937 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:29:54.905879    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34d15073e4fb036a626585eb9e65a9226946d65ef1d3ab87b67fef6f1edad6eb"
Nov 03 21:30:00.166692 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.166659    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp"]
Nov 03 21:30:00.167456 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.167433    5053 cpu_manager.go:414] "containerMap: removing container" podUID="309804de-c1da-49fc-96b3-53ba8be575fa" containerName="revoke-keys"
Nov 03 21:30:00.167562 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.167458    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="309804de-c1da-49fc-96b3-53ba8be575fa" containerName="revoke-keys"
Nov 03 21:30:00.167653 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.167640    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="309804de-c1da-49fc-96b3-53ba8be575fa" containerName="revoke-keys"
Nov 03 21:30:00.170825 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.170804    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8"]
Nov 03 21:30:00.170965 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.170941    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp"
Nov 03 21:30:00.174469 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.174449    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6"]
Nov 03 21:30:00.174585 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.174570    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8"
Nov 03 21:30:00.178034 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.178014    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6"
Nov 03 21:30:00.216374 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.216341    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4"]
Nov 03 21:30:00.220160 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.220140    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4"
Nov 03 21:30:00.222606 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.222580    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:30:00.222606 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.222606    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:30:00.222777 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.222680    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:30:00.256584 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.256550    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76s47\" (UniqueName: \"kubernetes.io/projected/144b733d-6d26-4755-860c-c96eb18ae726-kube-api-access-76s47\") pod \"maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6\" (UID: \"144b733d-6d26-4755-860c-c96eb18ae726\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6"
Nov 03 21:30:00.256789 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.256592    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7xmk\" (UniqueName: \"kubernetes.io/projected/98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7-kube-api-access-w7xmk\") pod \"maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp\" (UID: \"98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp"
Nov 03 21:30:00.256789 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.256615    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144b733d-6d26-4755-860c-c96eb18ae726-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6\" (UID: \"144b733d-6d26-4755-860c-c96eb18ae726\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6"
Nov 03 21:30:00.256789 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.256775    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/668e3b63-7725-43d5-b3df-739558563da8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8\" (UID: \"668e3b63-7725-43d5-b3df-739558563da8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8"
Nov 03 21:30:00.256958 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.256837    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcmlg\" (UniqueName: \"kubernetes.io/projected/e7818488-ea69-4147-985c-6869c69190d3-kube-api-access-vcmlg\") pod \"maas-api-key-cleanup-29895690-rqbx4\" (UID: \"e7818488-ea69-4147-985c-6869c69190d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4"
Nov 03 21:30:00.256958 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.256886    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7818488-ea69-4147-985c-6869c69190d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895690-rqbx4\" (UID: \"e7818488-ea69-4147-985c-6869c69190d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4"
Nov 03 21:30:00.257055 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.257011    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp\" (UID: \"98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp"
Nov 03 21:30:00.257103 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.257091    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd6x7\" (UniqueName: \"kubernetes.io/projected/668e3b63-7725-43d5-b3df-739558563da8-kube-api-access-rd6x7\") pod \"maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8\" (UID: \"668e3b63-7725-43d5-b3df-739558563da8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8"
Nov 03 21:30:00.269468 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.269432    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6"]
Nov 03 21:30:00.273037 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.272999    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp"]
Nov 03 21:30:00.275820 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.275799    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8"]
Nov 03 21:30:00.334093 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.334058    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4"]
Nov 03 21:30:00.358651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.358617    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76s47\" (UniqueName: \"kubernetes.io/projected/144b733d-6d26-4755-860c-c96eb18ae726-kube-api-access-76s47\") pod \"maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6\" (UID: \"144b733d-6d26-4755-860c-c96eb18ae726\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6"
Nov 03 21:30:00.358858 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.358655    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7xmk\" (UniqueName: \"kubernetes.io/projected/98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7-kube-api-access-w7xmk\") pod \"maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp\" (UID: \"98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp"
Nov 03 21:30:00.358858 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.358675    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144b733d-6d26-4755-860c-c96eb18ae726-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6\" (UID: \"144b733d-6d26-4755-860c-c96eb18ae726\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6"
Nov 03 21:30:00.358858 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.358726    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/668e3b63-7725-43d5-b3df-739558563da8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8\" (UID: \"668e3b63-7725-43d5-b3df-739558563da8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8"
Nov 03 21:30:00.358858 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.358772    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcmlg\" (UniqueName: \"kubernetes.io/projected/e7818488-ea69-4147-985c-6869c69190d3-kube-api-access-vcmlg\") pod \"maas-api-key-cleanup-29895690-rqbx4\" (UID: \"e7818488-ea69-4147-985c-6869c69190d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4"
Nov 03 21:30:00.358858 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.358812    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7818488-ea69-4147-985c-6869c69190d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895690-rqbx4\" (UID: \"e7818488-ea69-4147-985c-6869c69190d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4"
Nov 03 21:30:00.359181 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.358901    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp\" (UID: \"98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp"
Nov 03 21:30:00.359181 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.358973    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd6x7\" (UniqueName: \"kubernetes.io/projected/668e3b63-7725-43d5-b3df-739558563da8-kube-api-access-rd6x7\") pod \"maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8\" (UID: \"668e3b63-7725-43d5-b3df-739558563da8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8"
Nov 03 21:30:00.359516 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.359492    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144b733d-6d26-4755-860c-c96eb18ae726-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6\" (UID: \"144b733d-6d26-4755-860c-c96eb18ae726\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6"
Nov 03 21:30:00.359753 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.359727    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/668e3b63-7725-43d5-b3df-739558563da8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8\" (UID: \"668e3b63-7725-43d5-b3df-739558563da8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8"
Nov 03 21:30:00.359866 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.359849    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp\" (UID: \"98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp"
Nov 03 21:30:00.359958 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.359810    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7818488-ea69-4147-985c-6869c69190d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895690-rqbx4\" (UID: \"e7818488-ea69-4147-985c-6869c69190d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4"
Nov 03 21:30:00.415349 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.415320    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcmlg\" (UniqueName: \"kubernetes.io/projected/e7818488-ea69-4147-985c-6869c69190d3-kube-api-access-vcmlg\") pod \"maas-api-key-cleanup-29895690-rqbx4\" (UID: \"e7818488-ea69-4147-985c-6869c69190d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4"
Nov 03 21:30:00.415349 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.415339    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76s47\" (UniqueName: \"kubernetes.io/projected/144b733d-6d26-4755-860c-c96eb18ae726-kube-api-access-76s47\") pod \"maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6\" (UID: \"144b733d-6d26-4755-860c-c96eb18ae726\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6"
Nov 03 21:30:00.415564 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.415429    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd6x7\" (UniqueName: \"kubernetes.io/projected/668e3b63-7725-43d5-b3df-739558563da8-kube-api-access-rd6x7\") pod \"maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8\" (UID: \"668e3b63-7725-43d5-b3df-739558563da8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8"
Nov 03 21:30:00.415564 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.415484    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7xmk\" (UniqueName: \"kubernetes.io/projected/98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7-kube-api-access-w7xmk\") pod \"maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp\" (UID: \"98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp"
Nov 03 21:30:00.485844 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.485809    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp"
Nov 03 21:30:00.493055 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.493036    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8"
Nov 03 21:30:00.498965 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.498943    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6"
Nov 03 21:30:00.533152 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.533114    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4"
Nov 03 21:30:00.875679 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.875648    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp"]
Nov 03 21:30:00.877146 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:30:00.877123    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b9ffd1_ae5c_44d2_bdd8_acb41e4263f7.slice/crio-18e178e50a78e1d62b7c62d67869313710bd12191c7040f7dd6af528fd257576 WatchSource:0}: Error finding container 18e178e50a78e1d62b7c62d67869313710bd12191c7040f7dd6af528fd257576: Status 404 returned error can't find the container with id 18e178e50a78e1d62b7c62d67869313710bd12191c7040f7dd6af528fd257576
Nov 03 21:30:00.924502 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.924476    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8"]
Nov 03 21:30:00.925083 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:30:00.924972    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod668e3b63_7725_43d5_b3df_739558563da8.slice/crio-0c31d47f7d565a63683206e13890124c3c906770e7bc34590f03f7b30583f177 WatchSource:0}: Error finding container 0c31d47f7d565a63683206e13890124c3c906770e7bc34590f03f7b30583f177: Status 404 returned error can't find the container with id 0c31d47f7d565a63683206e13890124c3c906770e7bc34590f03f7b30583f177
Nov 03 21:30:00.927358 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.927331    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6"]
Nov 03 21:30:00.927944 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:30:00.927920    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144b733d_6d26_4755_860c_c96eb18ae726.slice/crio-a7d630a1a10cede9dc57bf53ff14414f0b4bbe857487a7a53dc1b2499e012641 WatchSource:0}: Error finding container a7d630a1a10cede9dc57bf53ff14414f0b4bbe857487a7a53dc1b2499e012641: Status 404 returned error can't find the container with id a7d630a1a10cede9dc57bf53ff14414f0b4bbe857487a7a53dc1b2499e012641
Nov 03 21:30:00.949477 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.949446    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6" event={"ID":"144b733d-6d26-4755-860c-c96eb18ae726","Type":"ContainerStarted","Data":"a7d630a1a10cede9dc57bf53ff14414f0b4bbe857487a7a53dc1b2499e012641"}
Nov 03 21:30:00.951001 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.950974    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8" event={"ID":"668e3b63-7725-43d5-b3df-739558563da8","Type":"ContainerStarted","Data":"0c31d47f7d565a63683206e13890124c3c906770e7bc34590f03f7b30583f177"}
Nov 03 21:30:00.952496 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.952474    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp" event={"ID":"98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7","Type":"ContainerStarted","Data":"18e178e50a78e1d62b7c62d67869313710bd12191c7040f7dd6af528fd257576"}
Nov 03 21:30:00.974144 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:00.974120    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4"]
Nov 03 21:30:00.975354 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:30:00.975332    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7818488_ea69_4147_985c_6869c69190d3.slice/crio-2b211ac8b721c2c02c7c6b4ecf6aec10f244e995a68af62b29f48c56c4682aba WatchSource:0}: Error finding container 2b211ac8b721c2c02c7c6b4ecf6aec10f244e995a68af62b29f48c56c4682aba: Status 404 returned error can't find the container with id 2b211ac8b721c2c02c7c6b4ecf6aec10f244e995a68af62b29f48c56c4682aba
Nov 03 21:30:01.965313 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:01.965273    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4" event={"ID":"e7818488-ea69-4147-985c-6869c69190d3","Type":"ContainerStarted","Data":"35ec1ad245700e8fc69fdbce2b8af7f2252447f09705a4b389f6226240fb751b"}
Nov 03 21:30:01.965781 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:01.965334    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4" event={"ID":"e7818488-ea69-4147-985c-6869c69190d3","Type":"ContainerStarted","Data":"2b211ac8b721c2c02c7c6b4ecf6aec10f244e995a68af62b29f48c56c4682aba"}
Nov 03 21:30:01.967000 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:01.966974    5053 generic.go:356] "Generic (PLEG): container finished" podID="144b733d-6d26-4755-860c-c96eb18ae726" containerID="021a852ed2f74342cb67bf0fb10b8afe62440507d089e88aad5367486eb9771f" exitCode=60
Nov 03 21:30:01.967122 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:01.967047    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6" event={"ID":"144b733d-6d26-4755-860c-c96eb18ae726","Type":"ContainerDied","Data":"021a852ed2f74342cb67bf0fb10b8afe62440507d089e88aad5367486eb9771f"}
Nov 03 21:30:01.967279 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:01.967253    5053 scope.go:122] "RemoveContainer" containerID="021a852ed2f74342cb67bf0fb10b8afe62440507d089e88aad5367486eb9771f"
Nov 03 21:30:01.968881 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:01.968855    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8" event={"ID":"668e3b63-7725-43d5-b3df-739558563da8","Type":"ContainerStarted","Data":"e373142cea892cbb63929907c03ee6fbe35c14b8f001ef7442f8e0383a64b7a1"}
Nov 03 21:30:01.970540 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:01.970514    5053 generic.go:356] "Generic (PLEG): container finished" podID="98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7" containerID="b3a5cc25ee3ed258b90b63365af3c5d39c19d2a5692c0909bb05097c14bca313" exitCode=60
Nov 03 21:30:01.970632 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:01.970544    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp" event={"ID":"98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7","Type":"ContainerDied","Data":"b3a5cc25ee3ed258b90b63365af3c5d39c19d2a5692c0909bb05097c14bca313"}
Nov 03 21:30:01.970804 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:01.970787    5053 scope.go:122] "RemoveContainer" containerID="b3a5cc25ee3ed258b90b63365af3c5d39c19d2a5692c0909bb05097c14bca313"
Nov 03 21:30:02.079831 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:02.079776    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4" podStartSLOduration=2.079759511 podStartE2EDuration="2.079759511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:02.076776022 +0000 UTC m=+916.570697898" watchObservedRunningTime="2026-11-03 21:30:02.079759511 +0000 UTC m=+916.573681387"
Nov 03 21:30:02.302678 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:02.302623    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8" podStartSLOduration=2.302605023 podStartE2EDuration="2.302605023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:02.300930817 +0000 UTC m=+916.794852693" watchObservedRunningTime="2026-11-03 21:30:02.302605023 +0000 UTC m=+916.796526899"
Nov 03 21:30:02.978507 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:02.978474    5053 generic.go:356] "Generic (PLEG): container finished" podID="e7818488-ea69-4147-985c-6869c69190d3" containerID="35ec1ad245700e8fc69fdbce2b8af7f2252447f09705a4b389f6226240fb751b" exitCode=60
Nov 03 21:30:02.978983 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:02.978574    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4" event={"ID":"e7818488-ea69-4147-985c-6869c69190d3","Type":"ContainerDied","Data":"35ec1ad245700e8fc69fdbce2b8af7f2252447f09705a4b389f6226240fb751b"}
Nov 03 21:30:02.978983 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:02.978843    5053 scope.go:122] "RemoveContainer" containerID="35ec1ad245700e8fc69fdbce2b8af7f2252447f09705a4b389f6226240fb751b"
Nov 03 21:30:02.981906 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:02.981875    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6" event={"ID":"144b733d-6d26-4755-860c-c96eb18ae726","Type":"ContainerStarted","Data":"1528d323498eb608b1f222c71a361bf5f8a78584ef10d8457e08e068bed2a8ed"}
Nov 03 21:30:02.984691 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:02.984663    5053 generic.go:356] "Generic (PLEG): container finished" podID="668e3b63-7725-43d5-b3df-739558563da8" containerID="e373142cea892cbb63929907c03ee6fbe35c14b8f001ef7442f8e0383a64b7a1" exitCode=60
Nov 03 21:30:02.984806 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:02.984749    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8" event={"ID":"668e3b63-7725-43d5-b3df-739558563da8","Type":"ContainerDied","Data":"e373142cea892cbb63929907c03ee6fbe35c14b8f001ef7442f8e0383a64b7a1"}
Nov 03 21:30:02.984968 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:02.984951    5053 scope.go:122] "RemoveContainer" containerID="e373142cea892cbb63929907c03ee6fbe35c14b8f001ef7442f8e0383a64b7a1"
Nov 03 21:30:02.987182 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:02.987159    5053 generic.go:356] "Generic (PLEG): container finished" podID="98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7" containerID="52f02dfb746f3d64b7ccf328340f5ae17dc62f133a78cfde6bfb4f9a121b996e" exitCode=60
Nov 03 21:30:02.987323 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:02.987211    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp" event={"ID":"98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7","Type":"ContainerDied","Data":"52f02dfb746f3d64b7ccf328340f5ae17dc62f133a78cfde6bfb4f9a121b996e"}
Nov 03 21:30:02.987323 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:02.987274    5053 scope.go:122] "RemoveContainer" containerID="b3a5cc25ee3ed258b90b63365af3c5d39c19d2a5692c0909bb05097c14bca313"
Nov 03 21:30:02.987516 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:02.987499    5053 scope.go:122] "RemoveContainer" containerID="52f02dfb746f3d64b7ccf328340f5ae17dc62f133a78cfde6bfb4f9a121b996e"
Nov 03 21:30:02.987776 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:30:02.987748    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp_odh-ai-gateway-infra(98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp" podUID="98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7"
Nov 03 21:30:03.427753 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:03.427709    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6" podStartSLOduration=3.427696538 podStartE2EDuration="3.427696538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:03.426628579 +0000 UTC m=+917.920550455" watchObservedRunningTime="2026-11-03 21:30:03.427696538 +0000 UTC m=+917.921618412"
Nov 03 21:30:03.995441 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:03.995407    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4" event={"ID":"e7818488-ea69-4147-985c-6869c69190d3","Type":"ContainerStarted","Data":"0d93e26959707fdc4f385249c32664d0200cb238143a95fafceae02035a98f33"}
Nov 03 21:30:03.997147 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:03.997121    5053 generic.go:356] "Generic (PLEG): container finished" podID="144b733d-6d26-4755-860c-c96eb18ae726" containerID="1528d323498eb608b1f222c71a361bf5f8a78584ef10d8457e08e068bed2a8ed" exitCode=60
Nov 03 21:30:03.997323 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:03.997202    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6" event={"ID":"144b733d-6d26-4755-860c-c96eb18ae726","Type":"ContainerDied","Data":"1528d323498eb608b1f222c71a361bf5f8a78584ef10d8457e08e068bed2a8ed"}
Nov 03 21:30:03.997323 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:03.997294    5053 scope.go:122] "RemoveContainer" containerID="021a852ed2f74342cb67bf0fb10b8afe62440507d089e88aad5367486eb9771f"
Nov 03 21:30:03.997462 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:03.997446    5053 scope.go:122] "RemoveContainer" containerID="1528d323498eb608b1f222c71a361bf5f8a78584ef10d8457e08e068bed2a8ed"
Nov 03 21:30:03.997721 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:30:03.997697    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6_odh-ai-gateway-infra(144b733d-6d26-4755-860c-c96eb18ae726)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6" podUID="144b733d-6d26-4755-860c-c96eb18ae726"
Nov 03 21:30:03.999230 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:03.999212    5053 generic.go:356] "Generic (PLEG): container finished" podID="668e3b63-7725-43d5-b3df-739558563da8" containerID="b63f919f34477c7257a54a0b20e4d5b122a588e7cf898adb083da45579c9118e" exitCode=60
Nov 03 21:30:03.999346 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:03.999267    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8" event={"ID":"668e3b63-7725-43d5-b3df-739558563da8","Type":"ContainerDied","Data":"b63f919f34477c7257a54a0b20e4d5b122a588e7cf898adb083da45579c9118e"}
Nov 03 21:30:03.999508 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:03.999496    5053 scope.go:122] "RemoveContainer" containerID="b63f919f34477c7257a54a0b20e4d5b122a588e7cf898adb083da45579c9118e"
Nov 03 21:30:03.999736 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:30:03.999719    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8_odh-ai-gateway-infra(668e3b63-7725-43d5-b3df-739558563da8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8" podUID="668e3b63-7725-43d5-b3df-739558563da8"
Nov 03 21:30:04.003534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:04.003512    5053 scope.go:122] "RemoveContainer" containerID="52f02dfb746f3d64b7ccf328340f5ae17dc62f133a78cfde6bfb4f9a121b996e"
Nov 03 21:30:04.003758 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:30:04.003736    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp_odh-ai-gateway-infra(98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp" podUID="98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7"
Nov 03 21:30:04.066345 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:04.066320    5053 scope.go:122] "RemoveContainer" containerID="e373142cea892cbb63929907c03ee6fbe35c14b8f001ef7442f8e0383a64b7a1"
Nov 03 21:30:05.015201 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:05.015164    5053 generic.go:356] "Generic (PLEG): container finished" podID="e7818488-ea69-4147-985c-6869c69190d3" containerID="0d93e26959707fdc4f385249c32664d0200cb238143a95fafceae02035a98f33" exitCode=60
Nov 03 21:30:05.015706 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:05.015256    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4" event={"ID":"e7818488-ea69-4147-985c-6869c69190d3","Type":"ContainerDied","Data":"0d93e26959707fdc4f385249c32664d0200cb238143a95fafceae02035a98f33"}
Nov 03 21:30:05.015706 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:05.015328    5053 scope.go:122] "RemoveContainer" containerID="35ec1ad245700e8fc69fdbce2b8af7f2252447f09705a4b389f6226240fb751b"
Nov 03 21:30:05.015706 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:05.015609    5053 scope.go:122] "RemoveContainer" containerID="0d93e26959707fdc4f385249c32664d0200cb238143a95fafceae02035a98f33"
Nov 03 21:30:05.015888 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:30:05.015867    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895690-rqbx4_odh-ai-gateway-infra(e7818488-ea69-4147-985c-6869c69190d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4" podUID="e7818488-ea69-4147-985c-6869c69190d3"
Nov 03 21:30:05.018194 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:05.018178    5053 scope.go:122] "RemoveContainer" containerID="1528d323498eb608b1f222c71a361bf5f8a78584ef10d8457e08e068bed2a8ed"
Nov 03 21:30:05.018457 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:30:05.018439    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6_odh-ai-gateway-infra(144b733d-6d26-4755-860c-c96eb18ae726)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6" podUID="144b733d-6d26-4755-860c-c96eb18ae726"
Nov 03 21:30:05.019628 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:05.019611    5053 scope.go:122] "RemoveContainer" containerID="b63f919f34477c7257a54a0b20e4d5b122a588e7cf898adb083da45579c9118e"
Nov 03 21:30:05.019799 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:30:05.019784    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8_odh-ai-gateway-infra(668e3b63-7725-43d5-b3df-739558563da8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8" podUID="668e3b63-7725-43d5-b3df-739558563da8"
Nov 03 21:30:06.027163 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:06.027132    5053 scope.go:122] "RemoveContainer" containerID="0d93e26959707fdc4f385249c32664d0200cb238143a95fafceae02035a98f33"
Nov 03 21:30:06.027591 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:30:06.027442    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895690-rqbx4_odh-ai-gateway-infra(e7818488-ea69-4147-985c-6869c69190d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4" podUID="e7818488-ea69-4147-985c-6869c69190d3"
Nov 03 21:30:10.285644 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:10.285610    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-012641b7-v8vr4"]
Nov 03 21:30:10.290852 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:10.290834    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-012641b7-v8vr4"
Nov 03 21:30:10.347659 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:10.347625    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-012641b7-v8vr4"]
Nov 03 21:30:10.383999 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:10.383967    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e216b0b7-4eef-456e-8f93-3d6c7dac25ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-012641b7-v8vr4\" (UID: \"e216b0b7-4eef-456e-8f93-3d6c7dac25ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012641b7-v8vr4"
Nov 03 21:30:10.384219 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:10.384039    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kwxt\" (UniqueName: \"kubernetes.io/projected/e216b0b7-4eef-456e-8f93-3d6c7dac25ad-kube-api-access-5kwxt\") pod \"maas-api-revoke-sub-012641b7-v8vr4\" (UID: \"e216b0b7-4eef-456e-8f93-3d6c7dac25ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012641b7-v8vr4"
Nov 03 21:30:10.384219 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:10.384182    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e216b0b7-4eef-456e-8f93-3d6c7dac25ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-012641b7-v8vr4\" (UID: \"e216b0b7-4eef-456e-8f93-3d6c7dac25ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012641b7-v8vr4"
Nov 03 21:30:10.391953 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:10.391923    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:30:10.485530 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:10.485494    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kwxt\" (UniqueName: \"kubernetes.io/projected/e216b0b7-4eef-456e-8f93-3d6c7dac25ad-kube-api-access-5kwxt\") pod \"maas-api-revoke-sub-012641b7-v8vr4\" (UID: \"e216b0b7-4eef-456e-8f93-3d6c7dac25ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012641b7-v8vr4"
Nov 03 21:30:10.485728 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:10.485555    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e216b0b7-4eef-456e-8f93-3d6c7dac25ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-012641b7-v8vr4\" (UID: \"e216b0b7-4eef-456e-8f93-3d6c7dac25ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012641b7-v8vr4"
Nov 03 21:30:10.485728 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:10.485638    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e216b0b7-4eef-456e-8f93-3d6c7dac25ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-012641b7-v8vr4\" (UID: \"e216b0b7-4eef-456e-8f93-3d6c7dac25ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012641b7-v8vr4"
Nov 03 21:30:10.486284 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:10.486255    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e216b0b7-4eef-456e-8f93-3d6c7dac25ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-012641b7-v8vr4\" (UID: \"e216b0b7-4eef-456e-8f93-3d6c7dac25ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012641b7-v8vr4"
Nov 03 21:30:10.488084 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:10.488056    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e216b0b7-4eef-456e-8f93-3d6c7dac25ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-012641b7-v8vr4\" (UID: \"e216b0b7-4eef-456e-8f93-3d6c7dac25ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012641b7-v8vr4"
Nov 03 21:30:10.542361 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:10.542292    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kwxt\" (UniqueName: \"kubernetes.io/projected/e216b0b7-4eef-456e-8f93-3d6c7dac25ad-kube-api-access-5kwxt\") pod \"maas-api-revoke-sub-012641b7-v8vr4\" (UID: \"e216b0b7-4eef-456e-8f93-3d6c7dac25ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012641b7-v8vr4"
Nov 03 21:30:10.603874 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:10.603840    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-012641b7-v8vr4"
Nov 03 21:30:10.928275 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:30:10.928169    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:30:10.982573 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:30:10.982526    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode216b0b7_4eef_456e_8f93_3d6c7dac25ad.slice/crio-e05c2dab0987fdeb1e68fc583e49e9f71e0c37dca9984bbcd54e0fabdc4efb70 WatchSource:0}: Error finding container e05c2dab0987fdeb1e68fc583e49e9f71e0c37dca9984bbcd54e0fabdc4efb70: Status 404 returned error can't find the container with id e05c2dab0987fdeb1e68fc583e49e9f71e0c37dca9984bbcd54e0fabdc4efb70
Nov 03 21:30:10.982886 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:10.982861    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-012641b7-v8vr4"]
Nov 03 21:30:11.060195 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:11.060164    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-012641b7-v8vr4" event={"ID":"e216b0b7-4eef-456e-8f93-3d6c7dac25ad","Type":"ContainerStarted","Data":"e05c2dab0987fdeb1e68fc583e49e9f71e0c37dca9984bbcd54e0fabdc4efb70"}
Nov 03 21:30:12.067711 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:12.067620    5053 generic.go:356] "Generic (PLEG): container finished" podID="e216b0b7-4eef-456e-8f93-3d6c7dac25ad" containerID="7fd955a21c0de66ceb8e93c7cb0a74089d84589fec1e4a747f9b54bfccb5de28" exitCode=0
Nov 03 21:30:12.068084 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:12.067708    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-012641b7-v8vr4" event={"ID":"e216b0b7-4eef-456e-8f93-3d6c7dac25ad","Type":"ContainerDied","Data":"7fd955a21c0de66ceb8e93c7cb0a74089d84589fec1e4a747f9b54bfccb5de28"}
Nov 03 21:30:13.239923 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:13.239898    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-012641b7-v8vr4"
Nov 03 21:30:13.318169 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:13.318141    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e216b0b7-4eef-456e-8f93-3d6c7dac25ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e216b0b7-4eef-456e-8f93-3d6c7dac25ad-maas-api-openshift-ca\") pod \"e216b0b7-4eef-456e-8f93-3d6c7dac25ad\" (UID: \"e216b0b7-4eef-456e-8f93-3d6c7dac25ad\") "
Nov 03 21:30:13.318400 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:13.318220    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e216b0b7-4eef-456e-8f93-3d6c7dac25ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e216b0b7-4eef-456e-8f93-3d6c7dac25ad-maas-api-xks-ca\") pod \"e216b0b7-4eef-456e-8f93-3d6c7dac25ad\" (UID: \"e216b0b7-4eef-456e-8f93-3d6c7dac25ad\") "
Nov 03 21:30:13.318400 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:13.318312    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e216b0b7-4eef-456e-8f93-3d6c7dac25ad-kube-api-access-5kwxt\" (UniqueName: \"kubernetes.io/projected/e216b0b7-4eef-456e-8f93-3d6c7dac25ad-kube-api-access-5kwxt\") pod \"e216b0b7-4eef-456e-8f93-3d6c7dac25ad\" (UID: \"e216b0b7-4eef-456e-8f93-3d6c7dac25ad\") "
Nov 03 21:30:13.318585 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:13.318561    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e216b0b7-4eef-456e-8f93-3d6c7dac25ad-maas-api-openshift-ca" pod "e216b0b7-4eef-456e-8f93-3d6c7dac25ad" (UID: "e216b0b7-4eef-456e-8f93-3d6c7dac25ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:30:13.318765 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:13.318750    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e216b0b7-4eef-456e-8f93-3d6c7dac25ad-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:13.320514 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:13.320489    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e216b0b7-4eef-456e-8f93-3d6c7dac25ad-maas-api-xks-ca" pod "e216b0b7-4eef-456e-8f93-3d6c7dac25ad" (UID: "e216b0b7-4eef-456e-8f93-3d6c7dac25ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:30:13.320594 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:13.320553    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e216b0b7-4eef-456e-8f93-3d6c7dac25ad-kube-api-access-5kwxt" pod "e216b0b7-4eef-456e-8f93-3d6c7dac25ad" (UID: "e216b0b7-4eef-456e-8f93-3d6c7dac25ad"). InnerVolumeSpecName "kube-api-access-5kwxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:30:13.420118 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:13.420084    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e216b0b7-4eef-456e-8f93-3d6c7dac25ad-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:13.420118 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:13.420119    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kwxt\" (UniqueName: \"kubernetes.io/projected/e216b0b7-4eef-456e-8f93-3d6c7dac25ad-kube-api-access-5kwxt\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:14.082528 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:14.082494    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-012641b7-v8vr4" event={"ID":"e216b0b7-4eef-456e-8f93-3d6c7dac25ad","Type":"ContainerDied","Data":"e05c2dab0987fdeb1e68fc583e49e9f71e0c37dca9984bbcd54e0fabdc4efb70"}
Nov 03 21:30:14.082740 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:14.082542    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e05c2dab0987fdeb1e68fc583e49e9f71e0c37dca9984bbcd54e0fabdc4efb70"
Nov 03 21:30:14.082740 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:14.082592    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-012641b7-v8vr4"
Nov 03 21:30:14.670263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:14.663167    5053 scope.go:122] "RemoveContainer" containerID="0d93e26959707fdc4f385249c32664d0200cb238143a95fafceae02035a98f33"
Nov 03 21:30:14.670263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:14.664703    5053 scope.go:122] "RemoveContainer" containerID="1528d323498eb608b1f222c71a361bf5f8a78584ef10d8457e08e068bed2a8ed"
Nov 03 21:30:16.096106 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:16.096015    5053 generic.go:356] "Generic (PLEG): container finished" podID="e7818488-ea69-4147-985c-6869c69190d3" containerID="f1bc40b5c681a3140b997a77b783ec318a8d366db77422b47494a1ebf6a16faf" exitCode=60
Nov 03 21:30:16.096106 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:16.096088    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4" event={"ID":"e7818488-ea69-4147-985c-6869c69190d3","Type":"ContainerDied","Data":"f1bc40b5c681a3140b997a77b783ec318a8d366db77422b47494a1ebf6a16faf"}
Nov 03 21:30:16.096679 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:16.096145    5053 scope.go:122] "RemoveContainer" containerID="0d93e26959707fdc4f385249c32664d0200cb238143a95fafceae02035a98f33"
Nov 03 21:30:16.096679 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:16.096569    5053 scope.go:122] "RemoveContainer" containerID="f1bc40b5c681a3140b997a77b783ec318a8d366db77422b47494a1ebf6a16faf"
Nov 03 21:30:16.097197 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:30:16.097055    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895690-rqbx4_odh-ai-gateway-infra(e7818488-ea69-4147-985c-6869c69190d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4" podUID="e7818488-ea69-4147-985c-6869c69190d3"
Nov 03 21:30:16.098396 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:16.098375    5053 generic.go:356] "Generic (PLEG): container finished" podID="144b733d-6d26-4755-860c-c96eb18ae726" containerID="634de4da166896526b432ef045ad41a0677f7e9145d61fb1befbc10f9585afa4" exitCode=60
Nov 03 21:30:16.098492 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:16.098468    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6" event={"ID":"144b733d-6d26-4755-860c-c96eb18ae726","Type":"ContainerDied","Data":"634de4da166896526b432ef045ad41a0677f7e9145d61fb1befbc10f9585afa4"}
Nov 03 21:30:16.098833 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:16.098816    5053 scope.go:122] "RemoveContainer" containerID="634de4da166896526b432ef045ad41a0677f7e9145d61fb1befbc10f9585afa4"
Nov 03 21:30:16.099110 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:30:16.099084    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6_odh-ai-gateway-infra(144b733d-6d26-4755-860c-c96eb18ae726)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6" podUID="144b733d-6d26-4755-860c-c96eb18ae726"
Nov 03 21:30:16.109846 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:16.109831    5053 scope.go:122] "RemoveContainer" containerID="1528d323498eb608b1f222c71a361bf5f8a78584ef10d8457e08e068bed2a8ed"
Nov 03 21:30:16.659402 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:16.659376    5053 scope.go:122] "RemoveContainer" containerID="b63f919f34477c7257a54a0b20e4d5b122a588e7cf898adb083da45579c9118e"
Nov 03 21:30:17.109859 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:17.109829    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8" event={"ID":"668e3b63-7725-43d5-b3df-739558563da8","Type":"ContainerStarted","Data":"5394081a37af44aafdccab1178d06e376169e75f1531b909011746438ee0c5a3"}
Nov 03 21:30:17.219324 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:17.219294    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4"]
Nov 03 21:30:17.370472 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:17.370443    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6"]
Nov 03 21:30:17.381619 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:17.381595    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4"
Nov 03 21:30:17.387913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:17.387887    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7818488-ea69-4147-985c-6869c69190d3-kube-api-access-vcmlg\" (UniqueName: \"kubernetes.io/projected/e7818488-ea69-4147-985c-6869c69190d3-kube-api-access-vcmlg\") pod \"e7818488-ea69-4147-985c-6869c69190d3\" (UID: \"e7818488-ea69-4147-985c-6869c69190d3\") "
Nov 03 21:30:17.388047 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:17.387959    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7818488-ea69-4147-985c-6869c69190d3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7818488-ea69-4147-985c-6869c69190d3-maas-api-service-ca\") pod \"e7818488-ea69-4147-985c-6869c69190d3\" (UID: \"e7818488-ea69-4147-985c-6869c69190d3\") "
Nov 03 21:30:17.388311 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:17.388290    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7818488-ea69-4147-985c-6869c69190d3-maas-api-service-ca" pod "e7818488-ea69-4147-985c-6869c69190d3" (UID: "e7818488-ea69-4147-985c-6869c69190d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:30:17.388427 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:17.388411    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7818488-ea69-4147-985c-6869c69190d3-maas-api-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:17.390282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:17.390218    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7818488-ea69-4147-985c-6869c69190d3-kube-api-access-vcmlg" pod "e7818488-ea69-4147-985c-6869c69190d3" (UID: "e7818488-ea69-4147-985c-6869c69190d3"). InnerVolumeSpecName "kube-api-access-vcmlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:30:17.489518 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:17.489496    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcmlg\" (UniqueName: \"kubernetes.io/projected/e7818488-ea69-4147-985c-6869c69190d3-kube-api-access-vcmlg\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:17.509207 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:17.509186    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6"
Nov 03 21:30:17.590081 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:17.590049    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/144b733d-6d26-4755-860c-c96eb18ae726-kube-api-access-76s47\" (UniqueName: \"kubernetes.io/projected/144b733d-6d26-4755-860c-c96eb18ae726-kube-api-access-76s47\") pod \"144b733d-6d26-4755-860c-c96eb18ae726\" (UID: \"144b733d-6d26-4755-860c-c96eb18ae726\") "
Nov 03 21:30:17.592170 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:17.592148    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144b733d-6d26-4755-860c-c96eb18ae726-kube-api-access-76s47" pod "144b733d-6d26-4755-860c-c96eb18ae726" (UID: "144b733d-6d26-4755-860c-c96eb18ae726"). InnerVolumeSpecName "kube-api-access-76s47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:30:17.659694 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:17.659669    5053 scope.go:122] "RemoveContainer" containerID="52f02dfb746f3d64b7ccf328340f5ae17dc62f133a78cfde6bfb4f9a121b996e"
Nov 03 21:30:17.690890 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:17.690861    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/144b733d-6d26-4755-860c-c96eb18ae726-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144b733d-6d26-4755-860c-c96eb18ae726-maas-api-service-ca\") pod \"144b733d-6d26-4755-860c-c96eb18ae726\" (UID: \"144b733d-6d26-4755-860c-c96eb18ae726\") "
Nov 03 21:30:17.691191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:17.691168    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144b733d-6d26-4755-860c-c96eb18ae726-maas-api-service-ca" pod "144b733d-6d26-4755-860c-c96eb18ae726" (UID: "144b733d-6d26-4755-860c-c96eb18ae726"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:30:17.691321 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:17.691307    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144b733d-6d26-4755-860c-c96eb18ae726-maas-api-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:17.691368 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:17.691326    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76s47\" (UniqueName: \"kubernetes.io/projected/144b733d-6d26-4755-860c-c96eb18ae726-kube-api-access-76s47\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:18.117731 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:18.117702    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4"
Nov 03 21:30:18.117731 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:18.117716    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4" event={"ID":"e7818488-ea69-4147-985c-6869c69190d3","Type":"ContainerDied","Data":"2b211ac8b721c2c02c7c6b4ecf6aec10f244e995a68af62b29f48c56c4682aba"}
Nov 03 21:30:18.118276 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:18.117817    5053 scope.go:122] "RemoveContainer" containerID="f1bc40b5c681a3140b997a77b783ec318a8d366db77422b47494a1ebf6a16faf"
Nov 03 21:30:18.119220 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:18.119199    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6"
Nov 03 21:30:18.119350 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:18.119250    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6" event={"ID":"144b733d-6d26-4755-860c-c96eb18ae726","Type":"ContainerDied","Data":"a7d630a1a10cede9dc57bf53ff14414f0b4bbe857487a7a53dc1b2499e012641"}
Nov 03 21:30:18.121633 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:18.121608    5053 generic.go:356] "Generic (PLEG): container finished" podID="668e3b63-7725-43d5-b3df-739558563da8" containerID="5394081a37af44aafdccab1178d06e376169e75f1531b909011746438ee0c5a3" exitCode=60
Nov 03 21:30:18.121716 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:18.121675    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8" event={"ID":"668e3b63-7725-43d5-b3df-739558563da8","Type":"ContainerDied","Data":"5394081a37af44aafdccab1178d06e376169e75f1531b909011746438ee0c5a3"}
Nov 03 21:30:18.122053 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:18.122032    5053 scope.go:122] "RemoveContainer" containerID="5394081a37af44aafdccab1178d06e376169e75f1531b909011746438ee0c5a3"
Nov 03 21:30:18.122307 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:30:18.122283    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8_odh-ai-gateway-infra(668e3b63-7725-43d5-b3df-739558563da8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8" podUID="668e3b63-7725-43d5-b3df-739558563da8"
Nov 03 21:30:18.123876 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:18.123858    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp" event={"ID":"98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7","Type":"ContainerStarted","Data":"c461a9cf0798c5a39452367a6162fdef54ef0c106bd94a786dbe1502417d65e6"}
Nov 03 21:30:18.129273 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:18.129228    5053 scope.go:122] "RemoveContainer" containerID="634de4da166896526b432ef045ad41a0677f7e9145d61fb1befbc10f9585afa4"
Nov 03 21:30:18.137957 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:18.137942    5053 scope.go:122] "RemoveContainer" containerID="b63f919f34477c7257a54a0b20e4d5b122a588e7cf898adb083da45579c9118e"
Nov 03 21:30:18.233207 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:18.233170    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8"]
Nov 03 21:30:18.282601 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:18.282561    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6"]
Nov 03 21:30:18.287103 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:18.287077    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895690-69cn6"]
Nov 03 21:30:18.398877 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:18.398827    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp" podStartSLOduration=18.398814554 podStartE2EDuration="18.398814554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:18.395880988 +0000 UTC m=+932.889802854" watchObservedRunningTime="2026-11-03 21:30:18.398814554 +0000 UTC m=+932.892736428"
Nov 03 21:30:18.623263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:18.623207    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4"]
Nov 03 21:30:18.627425 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:18.627395    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-rqbx4"]
Nov 03 21:30:19.133274 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:19.133210    5053 generic.go:356] "Generic (PLEG): container finished" podID="98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7" containerID="c461a9cf0798c5a39452367a6162fdef54ef0c106bd94a786dbe1502417d65e6" exitCode=60
Nov 03 21:30:19.133724 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:19.133274    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp" event={"ID":"98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7","Type":"ContainerDied","Data":"c461a9cf0798c5a39452367a6162fdef54ef0c106bd94a786dbe1502417d65e6"}
Nov 03 21:30:19.133724 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:19.133334    5053 scope.go:122] "RemoveContainer" containerID="52f02dfb746f3d64b7ccf328340f5ae17dc62f133a78cfde6bfb4f9a121b996e"
Nov 03 21:30:19.133724 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:19.133714    5053 scope.go:122] "RemoveContainer" containerID="c461a9cf0798c5a39452367a6162fdef54ef0c106bd94a786dbe1502417d65e6"
Nov 03 21:30:19.134061 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:30:19.134038    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp_odh-ai-gateway-infra(98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp" podUID="98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7"
Nov 03 21:30:19.297547 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:19.297518    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8"
Nov 03 21:30:19.308110 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:19.308083    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/668e3b63-7725-43d5-b3df-739558563da8-kube-api-access-rd6x7\" (UniqueName: \"kubernetes.io/projected/668e3b63-7725-43d5-b3df-739558563da8-kube-api-access-rd6x7\") pod \"668e3b63-7725-43d5-b3df-739558563da8\" (UID: \"668e3b63-7725-43d5-b3df-739558563da8\") "
Nov 03 21:30:19.308275 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:19.308208    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/668e3b63-7725-43d5-b3df-739558563da8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/668e3b63-7725-43d5-b3df-739558563da8-maas-api-service-ca\") pod \"668e3b63-7725-43d5-b3df-739558563da8\" (UID: \"668e3b63-7725-43d5-b3df-739558563da8\") "
Nov 03 21:30:19.308584 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:19.308556    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/668e3b63-7725-43d5-b3df-739558563da8-maas-api-service-ca" pod "668e3b63-7725-43d5-b3df-739558563da8" (UID: "668e3b63-7725-43d5-b3df-739558563da8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:30:19.308698 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:19.308639    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/668e3b63-7725-43d5-b3df-739558563da8-maas-api-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:19.310587 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:19.310565    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668e3b63-7725-43d5-b3df-739558563da8-kube-api-access-rd6x7" pod "668e3b63-7725-43d5-b3df-739558563da8" (UID: "668e3b63-7725-43d5-b3df-739558563da8"). InnerVolumeSpecName "kube-api-access-rd6x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:30:19.408750 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:19.408657    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp"]
Nov 03 21:30:19.409263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:19.409224    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd6x7\" (UniqueName: \"kubernetes.io/projected/668e3b63-7725-43d5-b3df-739558563da8-kube-api-access-rd6x7\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:19.666473 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:19.666389    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="144b733d-6d26-4755-860c-c96eb18ae726" path="/var/lib/kubelet/pods/144b733d-6d26-4755-860c-c96eb18ae726/volumes"
Nov 03 21:30:19.666801 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:19.666785    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7818488-ea69-4147-985c-6869c69190d3" path="/var/lib/kubelet/pods/e7818488-ea69-4147-985c-6869c69190d3/volumes"
Nov 03 21:30:20.145193 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:20.145165    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8"
Nov 03 21:30:20.145193 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:20.145176    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8" event={"ID":"668e3b63-7725-43d5-b3df-739558563da8","Type":"ContainerDied","Data":"0c31d47f7d565a63683206e13890124c3c906770e7bc34590f03f7b30583f177"}
Nov 03 21:30:20.145773 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:20.145287    5053 scope.go:122] "RemoveContainer" containerID="5394081a37af44aafdccab1178d06e376169e75f1531b909011746438ee0c5a3"
Nov 03 21:30:20.306871 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:20.306846    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp"
Nov 03 21:30:20.318254 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:20.318207    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7-maas-api-service-ca\") pod \"98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7\" (UID: \"98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7\") "
Nov 03 21:30:20.318394 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:20.318346    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7-kube-api-access-w7xmk\" (UniqueName: \"kubernetes.io/projected/98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7-kube-api-access-w7xmk\") pod \"98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7\" (UID: \"98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7\") "
Nov 03 21:30:20.318489 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:20.318469    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7-maas-api-service-ca" pod "98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7" (UID: "98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:30:20.318775 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:20.318758    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7-maas-api-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:20.320488 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:20.320468    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7-kube-api-access-w7xmk" pod "98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7" (UID: "98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7"). InnerVolumeSpecName "kube-api-access-w7xmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:30:20.418753 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:20.418671    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8"]
Nov 03 21:30:20.420322 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:20.420303    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7xmk\" (UniqueName: \"kubernetes.io/projected/98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7-kube-api-access-w7xmk\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:20.425815 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:20.425788    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895690-2qnp8"]
Nov 03 21:30:21.156020 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:21.155989    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp"
Nov 03 21:30:21.156020 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:21.156003    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp" event={"ID":"98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7","Type":"ContainerDied","Data":"18e178e50a78e1d62b7c62d67869313710bd12191c7040f7dd6af528fd257576"}
Nov 03 21:30:21.156530 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:21.156057    5053 scope.go:122] "RemoveContainer" containerID="c461a9cf0798c5a39452367a6162fdef54ef0c106bd94a786dbe1502417d65e6"
Nov 03 21:30:21.318690 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:21.318656    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp"]
Nov 03 21:30:21.368030 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:21.368000    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895690-vknwp"]
Nov 03 21:30:21.666495 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:21.666464    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="668e3b63-7725-43d5-b3df-739558563da8" path="/var/lib/kubelet/pods/668e3b63-7725-43d5-b3df-739558563da8/volumes"
Nov 03 21:30:21.666856 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:21.666844    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7" path="/var/lib/kubelet/pods/98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7/volumes"
Nov 03 21:30:25.832777 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.832697    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6057588-b6cjm"]
Nov 03 21:30:25.833287 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833269    5053 cpu_manager.go:414] "containerMap: removing container" podUID="e216b0b7-4eef-456e-8f93-3d6c7dac25ad" containerName="revoke-keys"
Nov 03 21:30:25.833342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833289    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e216b0b7-4eef-456e-8f93-3d6c7dac25ad" containerName="revoke-keys"
Nov 03 21:30:25.833342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833303    5053 cpu_manager.go:414] "containerMap: removing container" podUID="144b733d-6d26-4755-860c-c96eb18ae726" containerName="cleanup"
Nov 03 21:30:25.833342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833308    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144b733d-6d26-4755-860c-c96eb18ae726" containerName="cleanup"
Nov 03 21:30:25.833342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833314    5053 cpu_manager.go:414] "containerMap: removing container" podUID="144b733d-6d26-4755-860c-c96eb18ae726" containerName="cleanup"
Nov 03 21:30:25.833342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833319    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144b733d-6d26-4755-860c-c96eb18ae726" containerName="cleanup"
Nov 03 21:30:25.833342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833329    5053 cpu_manager.go:414] "containerMap: removing container" podUID="e7818488-ea69-4147-985c-6869c69190d3" containerName="cleanup"
Nov 03 21:30:25.833342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833334    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7818488-ea69-4147-985c-6869c69190d3" containerName="cleanup"
Nov 03 21:30:25.833534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833348    5053 cpu_manager.go:414] "containerMap: removing container" podUID="668e3b63-7725-43d5-b3df-739558563da8" containerName="cleanup"
Nov 03 21:30:25.833534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833356    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="668e3b63-7725-43d5-b3df-739558563da8" containerName="cleanup"
Nov 03 21:30:25.833534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833369    5053 cpu_manager.go:414] "containerMap: removing container" podUID="98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7" containerName="cleanup"
Nov 03 21:30:25.833534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833373    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7" containerName="cleanup"
Nov 03 21:30:25.833534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833385    5053 cpu_manager.go:414] "containerMap: removing container" podUID="668e3b63-7725-43d5-b3df-739558563da8" containerName="cleanup"
Nov 03 21:30:25.833534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833389    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="668e3b63-7725-43d5-b3df-739558563da8" containerName="cleanup"
Nov 03 21:30:25.833534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833413    5053 cpu_manager.go:414] "containerMap: removing container" podUID="98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7" containerName="cleanup"
Nov 03 21:30:25.833534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833417    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7" containerName="cleanup"
Nov 03 21:30:25.833534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833422    5053 cpu_manager.go:414] "containerMap: removing container" podUID="e7818488-ea69-4147-985c-6869c69190d3" containerName="cleanup"
Nov 03 21:30:25.833534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833427    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7818488-ea69-4147-985c-6869c69190d3" containerName="cleanup"
Nov 03 21:30:25.833534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833504    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7" containerName="cleanup"
Nov 03 21:30:25.833534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833512    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7818488-ea69-4147-985c-6869c69190d3" containerName="cleanup"
Nov 03 21:30:25.833534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833521    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="e216b0b7-4eef-456e-8f93-3d6c7dac25ad" containerName="revoke-keys"
Nov 03 21:30:25.833534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833527    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="144b733d-6d26-4755-860c-c96eb18ae726" containerName="cleanup"
Nov 03 21:30:25.833534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833532    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="144b733d-6d26-4755-860c-c96eb18ae726" containerName="cleanup"
Nov 03 21:30:25.833534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833538    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7818488-ea69-4147-985c-6869c69190d3" containerName="cleanup"
Nov 03 21:30:25.834025 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833543    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="668e3b63-7725-43d5-b3df-739558563da8" containerName="cleanup"
Nov 03 21:30:25.834025 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833548    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7" containerName="cleanup"
Nov 03 21:30:25.834025 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.833553    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="668e3b63-7725-43d5-b3df-739558563da8" containerName="cleanup"
Nov 03 21:30:25.838529 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.838505    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6057588-b6cjm"
Nov 03 21:30:25.881630 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.881598    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc752551-d8bd-439b-8cae-eb0faaf1ecb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6057588-b6cjm\" (UID: \"fc752551-d8bd-439b-8cae-eb0faaf1ecb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6057588-b6cjm"
Nov 03 21:30:25.881630 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.881636    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc752551-d8bd-439b-8cae-eb0faaf1ecb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6057588-b6cjm\" (UID: \"fc752551-d8bd-439b-8cae-eb0faaf1ecb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6057588-b6cjm"
Nov 03 21:30:25.881894 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.881777    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmw4f\" (UniqueName: \"kubernetes.io/projected/fc752551-d8bd-439b-8cae-eb0faaf1ecb2-kube-api-access-tmw4f\") pod \"maas-api-revoke-sub-a6057588-b6cjm\" (UID: \"fc752551-d8bd-439b-8cae-eb0faaf1ecb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6057588-b6cjm"
Nov 03 21:30:25.892161 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.892131    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6057588-b6cjm"]
Nov 03 21:30:25.938342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.938301    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:30:25.938342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.938306    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:30:25.938543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.938308    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:30:25.939897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.939877    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:30:25.982625 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.982593    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc752551-d8bd-439b-8cae-eb0faaf1ecb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6057588-b6cjm\" (UID: \"fc752551-d8bd-439b-8cae-eb0faaf1ecb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6057588-b6cjm"
Nov 03 21:30:25.982826 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.982632    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc752551-d8bd-439b-8cae-eb0faaf1ecb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6057588-b6cjm\" (UID: \"fc752551-d8bd-439b-8cae-eb0faaf1ecb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6057588-b6cjm"
Nov 03 21:30:25.982826 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.982698    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmw4f\" (UniqueName: \"kubernetes.io/projected/fc752551-d8bd-439b-8cae-eb0faaf1ecb2-kube-api-access-tmw4f\") pod \"maas-api-revoke-sub-a6057588-b6cjm\" (UID: \"fc752551-d8bd-439b-8cae-eb0faaf1ecb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6057588-b6cjm"
Nov 03 21:30:25.983436 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.983387    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc752551-d8bd-439b-8cae-eb0faaf1ecb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6057588-b6cjm\" (UID: \"fc752551-d8bd-439b-8cae-eb0faaf1ecb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6057588-b6cjm"
Nov 03 21:30:25.985248 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:25.985204    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc752551-d8bd-439b-8cae-eb0faaf1ecb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6057588-b6cjm\" (UID: \"fc752551-d8bd-439b-8cae-eb0faaf1ecb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6057588-b6cjm"
Nov 03 21:30:26.039379 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:26.039351    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmw4f\" (UniqueName: \"kubernetes.io/projected/fc752551-d8bd-439b-8cae-eb0faaf1ecb2-kube-api-access-tmw4f\") pod \"maas-api-revoke-sub-a6057588-b6cjm\" (UID: \"fc752551-d8bd-439b-8cae-eb0faaf1ecb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6057588-b6cjm"
Nov 03 21:30:26.152878 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:26.152800    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6057588-b6cjm"
Nov 03 21:30:26.527506 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:26.527463    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6057588-b6cjm"]
Nov 03 21:30:26.529725 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:30:26.529695    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc752551_d8bd_439b_8cae_eb0faaf1ecb2.slice/crio-917c981a16bf4d6c27c035de0e731bf892165180de92df277b5baad8b81b29cc WatchSource:0}: Error finding container 917c981a16bf4d6c27c035de0e731bf892165180de92df277b5baad8b81b29cc: Status 404 returned error can't find the container with id 917c981a16bf4d6c27c035de0e731bf892165180de92df277b5baad8b81b29cc
Nov 03 21:30:27.198072 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:27.198034    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6057588-b6cjm" event={"ID":"fc752551-d8bd-439b-8cae-eb0faaf1ecb2","Type":"ContainerStarted","Data":"0096b3c5e66025f031095cdec7ad986e98128d73989981e5b61fec71b018187f"}
Nov 03 21:30:27.198530 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:27.198094    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6057588-b6cjm" event={"ID":"fc752551-d8bd-439b-8cae-eb0faaf1ecb2","Type":"ContainerStarted","Data":"917c981a16bf4d6c27c035de0e731bf892165180de92df277b5baad8b81b29cc"}
Nov 03 21:30:27.312692 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:27.312641    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6057588-b6cjm" podStartSLOduration=2.312626469 podStartE2EDuration="2.312626469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:27.311477458 +0000 UTC m=+941.805399335" watchObservedRunningTime="2026-11-03 21:30:27.312626469 +0000 UTC m=+941.806548344"
Nov 03 21:30:28.207090 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:28.207056    5053 generic.go:356] "Generic (PLEG): container finished" podID="fc752551-d8bd-439b-8cae-eb0faaf1ecb2" containerID="0096b3c5e66025f031095cdec7ad986e98128d73989981e5b61fec71b018187f" exitCode=0
Nov 03 21:30:28.207486 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:28.207145    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6057588-b6cjm" event={"ID":"fc752551-d8bd-439b-8cae-eb0faaf1ecb2","Type":"ContainerDied","Data":"0096b3c5e66025f031095cdec7ad986e98128d73989981e5b61fec71b018187f"}
Nov 03 21:30:29.353393 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:29.353367    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6057588-b6cjm"
Nov 03 21:30:29.418903 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:29.418874    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc752551-d8bd-439b-8cae-eb0faaf1ecb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc752551-d8bd-439b-8cae-eb0faaf1ecb2-maas-api-xks-ca\") pod \"fc752551-d8bd-439b-8cae-eb0faaf1ecb2\" (UID: \"fc752551-d8bd-439b-8cae-eb0faaf1ecb2\") "
Nov 03 21:30:29.419046 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:29.418974    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc752551-d8bd-439b-8cae-eb0faaf1ecb2-kube-api-access-tmw4f\" (UniqueName: \"kubernetes.io/projected/fc752551-d8bd-439b-8cae-eb0faaf1ecb2-kube-api-access-tmw4f\") pod \"fc752551-d8bd-439b-8cae-eb0faaf1ecb2\" (UID: \"fc752551-d8bd-439b-8cae-eb0faaf1ecb2\") "
Nov 03 21:30:29.419113 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:29.419069    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc752551-d8bd-439b-8cae-eb0faaf1ecb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc752551-d8bd-439b-8cae-eb0faaf1ecb2-maas-api-openshift-ca\") pod \"fc752551-d8bd-439b-8cae-eb0faaf1ecb2\" (UID: \"fc752551-d8bd-439b-8cae-eb0faaf1ecb2\") "
Nov 03 21:30:29.419436 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:29.419410    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc752551-d8bd-439b-8cae-eb0faaf1ecb2-maas-api-openshift-ca" pod "fc752551-d8bd-439b-8cae-eb0faaf1ecb2" (UID: "fc752551-d8bd-439b-8cae-eb0faaf1ecb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:30:29.419547 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:29.419506    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc752551-d8bd-439b-8cae-eb0faaf1ecb2-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:29.420982 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:29.420964    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc752551-d8bd-439b-8cae-eb0faaf1ecb2-maas-api-xks-ca" pod "fc752551-d8bd-439b-8cae-eb0faaf1ecb2" (UID: "fc752551-d8bd-439b-8cae-eb0faaf1ecb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:30:29.421184 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:29.421172    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc752551-d8bd-439b-8cae-eb0faaf1ecb2-kube-api-access-tmw4f" pod "fc752551-d8bd-439b-8cae-eb0faaf1ecb2" (UID: "fc752551-d8bd-439b-8cae-eb0faaf1ecb2"). InnerVolumeSpecName "kube-api-access-tmw4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:30:29.520954 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:29.520878    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc752551-d8bd-439b-8cae-eb0faaf1ecb2-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:29.520954 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:29.520907    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmw4f\" (UniqueName: \"kubernetes.io/projected/fc752551-d8bd-439b-8cae-eb0faaf1ecb2-kube-api-access-tmw4f\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:30.221150 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.221120    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6057588-b6cjm"
Nov 03 21:30:30.221150 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.221130    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6057588-b6cjm" event={"ID":"fc752551-d8bd-439b-8cae-eb0faaf1ecb2","Type":"ContainerDied","Data":"917c981a16bf4d6c27c035de0e731bf892165180de92df277b5baad8b81b29cc"}
Nov 03 21:30:30.221414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.221200    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="917c981a16bf4d6c27c035de0e731bf892165180de92df277b5baad8b81b29cc"
Nov 03 21:30:30.580355 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.580266    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62a8bad8-dcxvw"]
Nov 03 21:30:30.580984 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.580963    5053 cpu_manager.go:414] "containerMap: removing container" podUID="98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7" containerName="cleanup"
Nov 03 21:30:30.581073 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.580986    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7" containerName="cleanup"
Nov 03 21:30:30.581073 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.581015    5053 cpu_manager.go:414] "containerMap: removing container" podUID="668e3b63-7725-43d5-b3df-739558563da8" containerName="cleanup"
Nov 03 21:30:30.581073 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.581023    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="668e3b63-7725-43d5-b3df-739558563da8" containerName="cleanup"
Nov 03 21:30:30.581073 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.581059    5053 cpu_manager.go:414] "containerMap: removing container" podUID="fc752551-d8bd-439b-8cae-eb0faaf1ecb2" containerName="revoke-keys"
Nov 03 21:30:30.581073 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.581066    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc752551-d8bd-439b-8cae-eb0faaf1ecb2" containerName="revoke-keys"
Nov 03 21:30:30.581352 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.581080    5053 cpu_manager.go:414] "containerMap: removing container" podUID="e7818488-ea69-4147-985c-6869c69190d3" containerName="cleanup"
Nov 03 21:30:30.581352 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.581087    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7818488-ea69-4147-985c-6869c69190d3" containerName="cleanup"
Nov 03 21:30:30.581352 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.581101    5053 cpu_manager.go:414] "containerMap: removing container" podUID="144b733d-6d26-4755-860c-c96eb18ae726" containerName="cleanup"
Nov 03 21:30:30.581352 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.581108    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144b733d-6d26-4755-860c-c96eb18ae726" containerName="cleanup"
Nov 03 21:30:30.581352 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.581265    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc752551-d8bd-439b-8cae-eb0faaf1ecb2" containerName="revoke-keys"
Nov 03 21:30:30.581352 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.581282    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="144b733d-6d26-4755-860c-c96eb18ae726" containerName="cleanup"
Nov 03 21:30:30.581352 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.581294    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7818488-ea69-4147-985c-6869c69190d3" containerName="cleanup"
Nov 03 21:30:30.581352 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.581307    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="98b9ffd1-ae5c-44d2-bdd8-acb41e4263f7" containerName="cleanup"
Nov 03 21:30:30.581352 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.581317    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="668e3b63-7725-43d5-b3df-739558563da8" containerName="cleanup"
Nov 03 21:30:30.584484 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.584465    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62a8bad8-dcxvw"
Nov 03 21:30:30.631328 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.631295    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjs6z\" (UniqueName: \"kubernetes.io/projected/827b72b2-5ec8-4186-9cba-e2c639fedde9-kube-api-access-jjs6z\") pod \"maas-api-revoke-sub-62a8bad8-dcxvw\" (UID: \"827b72b2-5ec8-4186-9cba-e2c639fedde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62a8bad8-dcxvw"
Nov 03 21:30:30.631513 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.631490    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/827b72b2-5ec8-4186-9cba-e2c639fedde9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62a8bad8-dcxvw\" (UID: \"827b72b2-5ec8-4186-9cba-e2c639fedde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62a8bad8-dcxvw"
Nov 03 21:30:30.631559 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.631539    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/827b72b2-5ec8-4186-9cba-e2c639fedde9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62a8bad8-dcxvw\" (UID: \"827b72b2-5ec8-4186-9cba-e2c639fedde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62a8bad8-dcxvw"
Nov 03 21:30:30.637514 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.637486    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:30:30.637514 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.637500    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:30:30.686677 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.686644    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:30:30.686677 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.686644    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:30:30.689523 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.689499    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62a8bad8-dcxvw"]
Nov 03 21:30:30.733148 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.733114    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/827b72b2-5ec8-4186-9cba-e2c639fedde9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62a8bad8-dcxvw\" (UID: \"827b72b2-5ec8-4186-9cba-e2c639fedde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62a8bad8-dcxvw"
Nov 03 21:30:30.733337 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.733163    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/827b72b2-5ec8-4186-9cba-e2c639fedde9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62a8bad8-dcxvw\" (UID: \"827b72b2-5ec8-4186-9cba-e2c639fedde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62a8bad8-dcxvw"
Nov 03 21:30:30.733337 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.733270    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjs6z\" (UniqueName: \"kubernetes.io/projected/827b72b2-5ec8-4186-9cba-e2c639fedde9-kube-api-access-jjs6z\") pod \"maas-api-revoke-sub-62a8bad8-dcxvw\" (UID: \"827b72b2-5ec8-4186-9cba-e2c639fedde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62a8bad8-dcxvw"
Nov 03 21:30:30.733804 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.733776    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/827b72b2-5ec8-4186-9cba-e2c639fedde9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62a8bad8-dcxvw\" (UID: \"827b72b2-5ec8-4186-9cba-e2c639fedde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62a8bad8-dcxvw"
Nov 03 21:30:30.735651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.735635    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/827b72b2-5ec8-4186-9cba-e2c639fedde9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62a8bad8-dcxvw\" (UID: \"827b72b2-5ec8-4186-9cba-e2c639fedde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62a8bad8-dcxvw"
Nov 03 21:30:30.789970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.789942    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjs6z\" (UniqueName: \"kubernetes.io/projected/827b72b2-5ec8-4186-9cba-e2c639fedde9-kube-api-access-jjs6z\") pod \"maas-api-revoke-sub-62a8bad8-dcxvw\" (UID: \"827b72b2-5ec8-4186-9cba-e2c639fedde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62a8bad8-dcxvw"
Nov 03 21:30:30.900924 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:30.900842    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62a8bad8-dcxvw"
Nov 03 21:30:31.284154 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:31.284123    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62a8bad8-dcxvw"]
Nov 03 21:30:31.284778 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:30:31.284752    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod827b72b2_5ec8_4186_9cba_e2c639fedde9.slice/crio-fe67c5f910f6e79b6df0f848c13762177e93ad7a3e24d6d365c128c4a86dc220 WatchSource:0}: Error finding container fe67c5f910f6e79b6df0f848c13762177e93ad7a3e24d6d365c128c4a86dc220: Status 404 returned error can't find the container with id fe67c5f910f6e79b6df0f848c13762177e93ad7a3e24d6d365c128c4a86dc220
Nov 03 21:30:32.240090 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:32.240052    5053 generic.go:356] "Generic (PLEG): container finished" podID="827b72b2-5ec8-4186-9cba-e2c639fedde9" containerID="7d116c38cbee2d838656a0a4d83495f2c8e68dc97f1a77f34b5edb9753fcf8f1" exitCode=0
Nov 03 21:30:32.240494 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:32.240136    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62a8bad8-dcxvw" event={"ID":"827b72b2-5ec8-4186-9cba-e2c639fedde9","Type":"ContainerDied","Data":"7d116c38cbee2d838656a0a4d83495f2c8e68dc97f1a77f34b5edb9753fcf8f1"}
Nov 03 21:30:32.240494 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:32.240201    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62a8bad8-dcxvw" event={"ID":"827b72b2-5ec8-4186-9cba-e2c639fedde9","Type":"ContainerStarted","Data":"fe67c5f910f6e79b6df0f848c13762177e93ad7a3e24d6d365c128c4a86dc220"}
Nov 03 21:30:33.425744 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:33.425712    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62a8bad8-dcxvw"
Nov 03 21:30:33.567701 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:33.567602    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/827b72b2-5ec8-4186-9cba-e2c639fedde9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/827b72b2-5ec8-4186-9cba-e2c639fedde9-maas-api-xks-ca\") pod \"827b72b2-5ec8-4186-9cba-e2c639fedde9\" (UID: \"827b72b2-5ec8-4186-9cba-e2c639fedde9\") "
Nov 03 21:30:33.567701 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:33.567669    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/827b72b2-5ec8-4186-9cba-e2c639fedde9-kube-api-access-jjs6z\" (UniqueName: \"kubernetes.io/projected/827b72b2-5ec8-4186-9cba-e2c639fedde9-kube-api-access-jjs6z\") pod \"827b72b2-5ec8-4186-9cba-e2c639fedde9\" (UID: \"827b72b2-5ec8-4186-9cba-e2c639fedde9\") "
Nov 03 21:30:33.567921 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:33.567748    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/827b72b2-5ec8-4186-9cba-e2c639fedde9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/827b72b2-5ec8-4186-9cba-e2c639fedde9-maas-api-openshift-ca\") pod \"827b72b2-5ec8-4186-9cba-e2c639fedde9\" (UID: \"827b72b2-5ec8-4186-9cba-e2c639fedde9\") "
Nov 03 21:30:33.576534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:33.576486    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/827b72b2-5ec8-4186-9cba-e2c639fedde9-maas-api-openshift-ca" pod "827b72b2-5ec8-4186-9cba-e2c639fedde9" (UID: "827b72b2-5ec8-4186-9cba-e2c639fedde9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:30:33.577728 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:33.577700    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/827b72b2-5ec8-4186-9cba-e2c639fedde9-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:33.583843 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:33.583809    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/827b72b2-5ec8-4186-9cba-e2c639fedde9-maas-api-xks-ca" pod "827b72b2-5ec8-4186-9cba-e2c639fedde9" (UID: "827b72b2-5ec8-4186-9cba-e2c639fedde9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:30:33.585103 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:33.585074    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/827b72b2-5ec8-4186-9cba-e2c639fedde9-kube-api-access-jjs6z" pod "827b72b2-5ec8-4186-9cba-e2c639fedde9" (UID: "827b72b2-5ec8-4186-9cba-e2c639fedde9"). InnerVolumeSpecName "kube-api-access-jjs6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:30:33.679600 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:33.679572    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/827b72b2-5ec8-4186-9cba-e2c639fedde9-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:33.679600 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:33.679600    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjs6z\" (UniqueName: \"kubernetes.io/projected/827b72b2-5ec8-4186-9cba-e2c639fedde9-kube-api-access-jjs6z\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:34.256436 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:34.256403    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62a8bad8-dcxvw"
Nov 03 21:30:34.256604 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:34.256432    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62a8bad8-dcxvw" event={"ID":"827b72b2-5ec8-4186-9cba-e2c639fedde9","Type":"ContainerDied","Data":"fe67c5f910f6e79b6df0f848c13762177e93ad7a3e24d6d365c128c4a86dc220"}
Nov 03 21:30:34.256604 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:34.256495    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe67c5f910f6e79b6df0f848c13762177e93ad7a3e24d6d365c128c4a86dc220"
Nov 03 21:30:40.928525 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:30:40.928489    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:30:41.330108 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:41.330072    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08d344e-q69kq"]
Nov 03 21:30:41.330928 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:41.330908    5053 cpu_manager.go:414] "containerMap: removing container" podUID="827b72b2-5ec8-4186-9cba-e2c639fedde9" containerName="revoke-keys"
Nov 03 21:30:41.331004 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:41.330932    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="827b72b2-5ec8-4186-9cba-e2c639fedde9" containerName="revoke-keys"
Nov 03 21:30:41.331110 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:41.331098    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="827b72b2-5ec8-4186-9cba-e2c639fedde9" containerName="revoke-keys"
Nov 03 21:30:41.335439 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:41.335420    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08d344e-q69kq"
Nov 03 21:30:41.388002 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:41.387974    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08d344e-q69kq"]
Nov 03 21:30:41.434258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:41.434207    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:30:41.434427 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:41.434270    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:30:41.434427 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:41.434279    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:30:41.434427 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:41.434279    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:30:41.469472 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:41.469438    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e467c2a-8216-45d3-b354-8c86d0931091-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f08d344e-q69kq\" (UID: \"5e467c2a-8216-45d3-b354-8c86d0931091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08d344e-q69kq"
Nov 03 21:30:41.469653 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:41.469580    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm7zm\" (UniqueName: \"kubernetes.io/projected/5e467c2a-8216-45d3-b354-8c86d0931091-kube-api-access-nm7zm\") pod \"maas-api-revoke-sub-f08d344e-q69kq\" (UID: \"5e467c2a-8216-45d3-b354-8c86d0931091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08d344e-q69kq"
Nov 03 21:30:41.469731 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:41.469659    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e467c2a-8216-45d3-b354-8c86d0931091-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f08d344e-q69kq\" (UID: \"5e467c2a-8216-45d3-b354-8c86d0931091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08d344e-q69kq"
Nov 03 21:30:41.570891 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:41.570854    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm7zm\" (UniqueName: \"kubernetes.io/projected/5e467c2a-8216-45d3-b354-8c86d0931091-kube-api-access-nm7zm\") pod \"maas-api-revoke-sub-f08d344e-q69kq\" (UID: \"5e467c2a-8216-45d3-b354-8c86d0931091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08d344e-q69kq"
Nov 03 21:30:41.571086 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:41.570900    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e467c2a-8216-45d3-b354-8c86d0931091-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f08d344e-q69kq\" (UID: \"5e467c2a-8216-45d3-b354-8c86d0931091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08d344e-q69kq"
Nov 03 21:30:41.571086 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:41.570976    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e467c2a-8216-45d3-b354-8c86d0931091-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f08d344e-q69kq\" (UID: \"5e467c2a-8216-45d3-b354-8c86d0931091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08d344e-q69kq"
Nov 03 21:30:41.571709 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:41.571690    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e467c2a-8216-45d3-b354-8c86d0931091-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f08d344e-q69kq\" (UID: \"5e467c2a-8216-45d3-b354-8c86d0931091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08d344e-q69kq"
Nov 03 21:30:41.573428 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:41.573409    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e467c2a-8216-45d3-b354-8c86d0931091-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f08d344e-q69kq\" (UID: \"5e467c2a-8216-45d3-b354-8c86d0931091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08d344e-q69kq"
Nov 03 21:30:41.627360 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:41.627285    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm7zm\" (UniqueName: \"kubernetes.io/projected/5e467c2a-8216-45d3-b354-8c86d0931091-kube-api-access-nm7zm\") pod \"maas-api-revoke-sub-f08d344e-q69kq\" (UID: \"5e467c2a-8216-45d3-b354-8c86d0931091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08d344e-q69kq"
Nov 03 21:30:41.650513 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:41.650485    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08d344e-q69kq"
Nov 03 21:30:42.028295 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.028261    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"]
Nov 03 21:30:42.033531 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.033512    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.034268 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.034215    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08d344e-q69kq"]
Nov 03 21:30:42.035955 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:30:42.035929    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e467c2a_8216_45d3_b354_8c86d0931091.slice/crio-ccdad11960a8b8967c62355917e62f0848626962e4211733c48c18de8c24602a WatchSource:0}: Error finding container ccdad11960a8b8967c62355917e62f0848626962e4211733c48c18de8c24602a: Status 404 returned error can't find the container with id ccdad11960a8b8967c62355917e62f0848626962e4211733c48c18de8c24602a
Nov 03 21:30:42.085157 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.085133    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7ee809-openshift-default-dockercfg-qsllw\""
Nov 03 21:30:42.123608 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.123579    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"]
Nov 03 21:30:42.176559 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.176525    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12490bb7-88e5-449b-82c1-5a315a521d36-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.176559 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.176563    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.176766 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.176589    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12490bb7-88e5-449b-82c1-5a315a521d36-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.176766 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.176715    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.176837 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.176768    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.176869 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.176847    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.176901 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.176883    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.176972 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.176953    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mjbz\" (UniqueName: \"kubernetes.io/projected/12490bb7-88e5-449b-82c1-5a315a521d36-kube-api-access-8mjbz\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.177014 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.177001    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12490bb7-88e5-449b-82c1-5a315a521d36-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.277709 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.277668    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mjbz\" (UniqueName: \"kubernetes.io/projected/12490bb7-88e5-449b-82c1-5a315a521d36-kube-api-access-8mjbz\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.277897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.277731    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12490bb7-88e5-449b-82c1-5a315a521d36-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.277897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.277753    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12490bb7-88e5-449b-82c1-5a315a521d36-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.277897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.277773    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.277897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.277790    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12490bb7-88e5-449b-82c1-5a315a521d36-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.277897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.277870    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.278167 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.277942    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.278167 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.278033    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.278167 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.278110    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.278373 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.278305    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.278443 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.278419    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.278508 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.278452    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.278562 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.278523    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.278640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.278614    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12490bb7-88e5-449b-82c1-5a315a521d36-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.280397 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.280379    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.280539 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.280520    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12490bb7-88e5-449b-82c1-5a315a521d36-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.314622 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.314596    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08d344e-q69kq" event={"ID":"5e467c2a-8216-45d3-b354-8c86d0931091","Type":"ContainerStarted","Data":"ccdad11960a8b8967c62355917e62f0848626962e4211733c48c18de8c24602a"}
Nov 03 21:30:42.334066 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.334038    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12490bb7-88e5-449b-82c1-5a315a521d36-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.334066 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.334050    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mjbz\" (UniqueName: \"kubernetes.io/projected/12490bb7-88e5-449b-82c1-5a315a521d36-kube-api-access-8mjbz\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.364036 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.364016    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:42.785043 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.785009    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"]
Nov 03 21:30:42.825309 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:30:42.825267    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12490bb7_88e5_449b_82c1_5a315a521d36.slice/crio-2b2cbc8d360daea82d707ee15489391410a82ac74b8b52403a1a17e093788aa7 WatchSource:0}: Error finding container 2b2cbc8d360daea82d707ee15489391410a82ac74b8b52403a1a17e093788aa7: Status 404 returned error can't find the container with id 2b2cbc8d360daea82d707ee15489391410a82ac74b8b52403a1a17e093788aa7
Nov 03 21:30:42.827838 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.827801    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:30:42.827948 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.827888    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:30:42.827948 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:42.827933    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:30:43.323018 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:43.322978    5053 generic.go:356] "Generic (PLEG): container finished" podID="5e467c2a-8216-45d3-b354-8c86d0931091" containerID="328d6ce386683033545bf1d052df3f9bf1825fcc4fd5bb1812a53faa8ddee48f" exitCode=0
Nov 03 21:30:43.323539 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:43.323063    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08d344e-q69kq" event={"ID":"5e467c2a-8216-45d3-b354-8c86d0931091","Type":"ContainerDied","Data":"328d6ce386683033545bf1d052df3f9bf1825fcc4fd5bb1812a53faa8ddee48f"}
Nov 03 21:30:43.324873 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:43.324851    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6" event={"ID":"12490bb7-88e5-449b-82c1-5a315a521d36","Type":"ContainerStarted","Data":"886e356ed56e3ce8bbcdbb526b5e3572b9f05d7a80920d883a2c5ab08643fa67"}
Nov 03 21:30:43.324967 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:43.324888    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6" event={"ID":"12490bb7-88e5-449b-82c1-5a315a521d36","Type":"ContainerStarted","Data":"2b2cbc8d360daea82d707ee15489391410a82ac74b8b52403a1a17e093788aa7"}
Nov 03 21:30:43.364390 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:43.364350    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:43.366008 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:43.365982    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 21:30:43.366120 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:43.366033    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6" podUID="12490bb7-88e5-449b-82c1-5a315a521d36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 21:30:43.553085 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:43.553027    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6" podStartSLOduration=2.553012728 podStartE2EDuration="2.553012728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:43.54960196 +0000 UTC m=+958.043523836" watchObservedRunningTime="2026-11-03 21:30:43.553012728 +0000 UTC m=+958.046934602"
Nov 03 21:30:44.364811 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:44.364781    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 21:30:44.365186 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:44.364832    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6" podUID="12490bb7-88e5-449b-82c1-5a315a521d36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 21:30:44.535722 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:44.535697    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08d344e-q69kq"
Nov 03 21:30:44.635739 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:44.635644    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e467c2a-8216-45d3-b354-8c86d0931091-kube-api-access-nm7zm\" (UniqueName: \"kubernetes.io/projected/5e467c2a-8216-45d3-b354-8c86d0931091-kube-api-access-nm7zm\") pod \"5e467c2a-8216-45d3-b354-8c86d0931091\" (UID: \"5e467c2a-8216-45d3-b354-8c86d0931091\") "
Nov 03 21:30:44.635739 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:44.635729    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e467c2a-8216-45d3-b354-8c86d0931091-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e467c2a-8216-45d3-b354-8c86d0931091-maas-api-xks-ca\") pod \"5e467c2a-8216-45d3-b354-8c86d0931091\" (UID: \"5e467c2a-8216-45d3-b354-8c86d0931091\") "
Nov 03 21:30:44.635984 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:44.635784    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e467c2a-8216-45d3-b354-8c86d0931091-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e467c2a-8216-45d3-b354-8c86d0931091-maas-api-openshift-ca\") pod \"5e467c2a-8216-45d3-b354-8c86d0931091\" (UID: \"5e467c2a-8216-45d3-b354-8c86d0931091\") "
Nov 03 21:30:44.636154 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:44.636124    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e467c2a-8216-45d3-b354-8c86d0931091-maas-api-openshift-ca" pod "5e467c2a-8216-45d3-b354-8c86d0931091" (UID: "5e467c2a-8216-45d3-b354-8c86d0931091"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:30:44.638111 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:44.638080    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e467c2a-8216-45d3-b354-8c86d0931091-maas-api-xks-ca" pod "5e467c2a-8216-45d3-b354-8c86d0931091" (UID: "5e467c2a-8216-45d3-b354-8c86d0931091"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:30:44.638212 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:44.638137    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e467c2a-8216-45d3-b354-8c86d0931091-kube-api-access-nm7zm" pod "5e467c2a-8216-45d3-b354-8c86d0931091" (UID: "5e467c2a-8216-45d3-b354-8c86d0931091"). InnerVolumeSpecName "kube-api-access-nm7zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:30:44.738076 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:44.737819    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e467c2a-8216-45d3-b354-8c86d0931091-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:44.738076 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:44.737851    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e467c2a-8216-45d3-b354-8c86d0931091-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:44.738076 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:44.737866    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm7zm\" (UniqueName: \"kubernetes.io/projected/5e467c2a-8216-45d3-b354-8c86d0931091-kube-api-access-nm7zm\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:45.343689 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.343661    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08d344e-q69kq"
Nov 03 21:30:45.344043 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.344009    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08d344e-q69kq" event={"ID":"5e467c2a-8216-45d3-b354-8c86d0931091","Type":"ContainerDied","Data":"ccdad11960a8b8967c62355917e62f0848626962e4211733c48c18de8c24602a"}
Nov 03 21:30:45.344193 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.344061    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccdad11960a8b8967c62355917e62f0848626962e4211733c48c18de8c24602a"
Nov 03 21:30:45.364688 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.364661    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 21:30:45.364833 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.364713    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6" podUID="12490bb7-88e5-449b-82c1-5a315a521d36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 21:30:45.465542 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.465502    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5"]
Nov 03 21:30:45.466161 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.466141    5053 cpu_manager.go:414] "containerMap: removing container" podUID="5e467c2a-8216-45d3-b354-8c86d0931091" containerName="revoke-keys"
Nov 03 21:30:45.466467 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.466163    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e467c2a-8216-45d3-b354-8c86d0931091" containerName="revoke-keys"
Nov 03 21:30:45.466467 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.466307    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e467c2a-8216-45d3-b354-8c86d0931091" containerName="revoke-keys"
Nov 03 21:30:45.469962 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.469940    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5"
Nov 03 21:30:45.524816 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.524462    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 21:30:45.524816 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.524583    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 21:30:45.524816 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.524720    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 21:30:45.524816 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.524779    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a14a3e029aa8\""
Nov 03 21:30:45.539608 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.539441    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 21:30:45.539608 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.539574    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-6bbcc490edc5\""
Nov 03 21:30:45.548056 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.547713    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5"]
Nov 03 21:30:45.557273 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.557188    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-tv44q\""
Nov 03 21:30:45.558597 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.558561    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-provider-credentials\") pod \"payload-processing-external-model-cb5b88ddd-qqqh5\" (UID: \"e2736c90-d7e3-4aa1-81dd-45dea43e3b3f\") " pod="models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5"
Nov 03 21:30:45.558860 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.558841    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbrv8\" (UniqueName: \"kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-kube-api-access-wbrv8\") pod \"payload-processing-external-model-cb5b88ddd-qqqh5\" (UID: \"e2736c90-d7e3-4aa1-81dd-45dea43e3b3f\") " pod="models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5"
Nov 03 21:30:45.559053 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.559035    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-plugins-config-volume\") pod \"payload-processing-external-model-cb5b88ddd-qqqh5\" (UID: \"e2736c90-d7e3-4aa1-81dd-45dea43e3b3f\") " pod="models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5"
Nov 03 21:30:45.559201 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.559181    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-routing-overlay-volume\") pod \"payload-processing-external-model-cb5b88ddd-qqqh5\" (UID: \"e2736c90-d7e3-4aa1-81dd-45dea43e3b3f\") " pod="models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5"
Nov 03 21:30:45.661112 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.661017    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-plugins-config-volume\") pod \"payload-processing-external-model-cb5b88ddd-qqqh5\" (UID: \"e2736c90-d7e3-4aa1-81dd-45dea43e3b3f\") " pod="models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5"
Nov 03 21:30:45.661112 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.661066    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-routing-overlay-volume\") pod \"payload-processing-external-model-cb5b88ddd-qqqh5\" (UID: \"e2736c90-d7e3-4aa1-81dd-45dea43e3b3f\") " pod="models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5"
Nov 03 21:30:45.661381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.661254    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-provider-credentials\") pod \"payload-processing-external-model-cb5b88ddd-qqqh5\" (UID: \"e2736c90-d7e3-4aa1-81dd-45dea43e3b3f\") " pod="models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5"
Nov 03 21:30:45.661381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.661307    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbrv8\" (UniqueName: \"kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-kube-api-access-wbrv8\") pod \"payload-processing-external-model-cb5b88ddd-qqqh5\" (UID: \"e2736c90-d7e3-4aa1-81dd-45dea43e3b3f\") " pod="models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5"
Nov 03 21:30:45.723263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.713532    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-6bbcc490edc5\""
Nov 03 21:30:45.723263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.713638    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 21:30:45.723263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.713758    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 21:30:45.723263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.714967    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-plugins-config-volume\") pod \"payload-processing-external-model-cb5b88ddd-qqqh5\" (UID: \"e2736c90-d7e3-4aa1-81dd-45dea43e3b3f\") " pod="models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5"
Nov 03 21:30:45.723263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.715390    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-routing-overlay-volume\") pod \"payload-processing-external-model-cb5b88ddd-qqqh5\" (UID: \"e2736c90-d7e3-4aa1-81dd-45dea43e3b3f\") " pod="models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5"
Nov 03 21:30:45.767178 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.767147    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 21:30:45.776074 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.776050    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a14a3e029aa8\""
Nov 03 21:30:45.795144 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.795110    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-provider-credentials\") pod \"payload-processing-external-model-cb5b88ddd-qqqh5\" (UID: \"e2736c90-d7e3-4aa1-81dd-45dea43e3b3f\") " pod="models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5"
Nov 03 21:30:45.836265 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.829518    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 21:30:45.844916 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:45.844883    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbrv8\" (UniqueName: \"kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-kube-api-access-wbrv8\") pod \"payload-processing-external-model-cb5b88ddd-qqqh5\" (UID: \"e2736c90-d7e3-4aa1-81dd-45dea43e3b3f\") " pod="models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5"
Nov 03 21:30:46.136550 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:46.136519    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-tv44q\""
Nov 03 21:30:46.145461 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:46.145434    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5"
Nov 03 21:30:46.365387 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:46.365347    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 21:30:46.365912 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:46.365429    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6" podUID="12490bb7-88e5-449b-82c1-5a315a521d36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 21:30:46.551377 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:46.550591    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5"]
Nov 03 21:30:46.552424 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:30:46.552377    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2736c90_d7e3_4aa1_81dd_45dea43e3b3f.slice/crio-894cbbb6a883ba7ce67b109d03da8603d8e76b17f083e77aabb468d6178cb138 WatchSource:0}: Error finding container 894cbbb6a883ba7ce67b109d03da8603d8e76b17f083e77aabb468d6178cb138: Status 404 returned error can't find the container with id 894cbbb6a883ba7ce67b109d03da8603d8e76b17f083e77aabb468d6178cb138
Nov 03 21:30:47.358318 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:47.358280    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5" event={"ID":"e2736c90-d7e3-4aa1-81dd-45dea43e3b3f","Type":"ContainerStarted","Data":"6bb05c759cb939604368e270cd6f62af0644e7e4d86e5eb613617a729114acb6"}
Nov 03 21:30:47.358536 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:47.358344    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5" event={"ID":"e2736c90-d7e3-4aa1-81dd-45dea43e3b3f","Type":"ContainerStarted","Data":"894cbbb6a883ba7ce67b109d03da8603d8e76b17f083e77aabb468d6178cb138"}
Nov 03 21:30:47.358536 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:47.358401    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5"
Nov 03 21:30:47.364932 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:47.364905    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 21:30:47.365008 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:47.364962    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6" podUID="12490bb7-88e5-449b-82c1-5a315a521d36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 21:30:47.478433 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:47.478373    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5" podStartSLOduration=2.478352563 podStartE2EDuration="2.478352563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:47.475209459 +0000 UTC m=+961.969131346" watchObservedRunningTime="2026-11-03 21:30:47.478352563 +0000 UTC m=+961.972274438"
Nov 03 21:30:48.364483 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:48.364448    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 21:30:48.364674 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:48.364491    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6" podUID="12490bb7-88e5-449b-82c1-5a315a521d36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 21:30:49.128913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:49.128872    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51eeb46f-qr5gs"]
Nov 03 21:30:49.133954 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:49.133930    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eeb46f-qr5gs"
Nov 03 21:30:49.186878 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:49.186791    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51eeb46f-qr5gs"]
Nov 03 21:30:49.201709 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:49.201658    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwc5v\" (UniqueName: \"kubernetes.io/projected/99a9cf09-1a48-4a77-9127-9ceb9279708e-kube-api-access-dwc5v\") pod \"maas-api-revoke-sub-51eeb46f-qr5gs\" (UID: \"99a9cf09-1a48-4a77-9127-9ceb9279708e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eeb46f-qr5gs"
Nov 03 21:30:49.201897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:49.201717    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a9cf09-1a48-4a77-9127-9ceb9279708e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51eeb46f-qr5gs\" (UID: \"99a9cf09-1a48-4a77-9127-9ceb9279708e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eeb46f-qr5gs"
Nov 03 21:30:49.201897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:49.201883    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a9cf09-1a48-4a77-9127-9ceb9279708e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51eeb46f-qr5gs\" (UID: \"99a9cf09-1a48-4a77-9127-9ceb9279708e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eeb46f-qr5gs"
Nov 03 21:30:49.232694 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:49.232662    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:30:49.233077 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:49.233047    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:30:49.235423 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:49.235393    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:30:49.235658 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:49.235618    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:30:49.303608 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:49.303566    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a9cf09-1a48-4a77-9127-9ceb9279708e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51eeb46f-qr5gs\" (UID: \"99a9cf09-1a48-4a77-9127-9ceb9279708e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eeb46f-qr5gs"
Nov 03 21:30:49.303813 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:49.303723    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwc5v\" (UniqueName: \"kubernetes.io/projected/99a9cf09-1a48-4a77-9127-9ceb9279708e-kube-api-access-dwc5v\") pod \"maas-api-revoke-sub-51eeb46f-qr5gs\" (UID: \"99a9cf09-1a48-4a77-9127-9ceb9279708e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eeb46f-qr5gs"
Nov 03 21:30:49.303813 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:49.303755    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a9cf09-1a48-4a77-9127-9ceb9279708e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51eeb46f-qr5gs\" (UID: \"99a9cf09-1a48-4a77-9127-9ceb9279708e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eeb46f-qr5gs"
Nov 03 21:30:49.304593 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:49.304564    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a9cf09-1a48-4a77-9127-9ceb9279708e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51eeb46f-qr5gs\" (UID: \"99a9cf09-1a48-4a77-9127-9ceb9279708e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eeb46f-qr5gs"
Nov 03 21:30:49.306473 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:49.306449    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a9cf09-1a48-4a77-9127-9ceb9279708e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51eeb46f-qr5gs\" (UID: \"99a9cf09-1a48-4a77-9127-9ceb9279708e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eeb46f-qr5gs"
Nov 03 21:30:49.360397 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:49.360362    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwc5v\" (UniqueName: \"kubernetes.io/projected/99a9cf09-1a48-4a77-9127-9ceb9279708e-kube-api-access-dwc5v\") pod \"maas-api-revoke-sub-51eeb46f-qr5gs\" (UID: \"99a9cf09-1a48-4a77-9127-9ceb9279708e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eeb46f-qr5gs"
Nov 03 21:30:49.453176 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:49.453147    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eeb46f-qr5gs"
Nov 03 21:30:49.460743 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:49.460709    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:49.461533 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:49.461513    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:49.462561 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:49.462546    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:30:49.836180 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:49.836156    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51eeb46f-qr5gs"]
Nov 03 21:30:49.838581 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:30:49.838548    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a9cf09_1a48_4a77_9127_9ceb9279708e.slice/crio-524f8c2133c54d5b95bcdd938af3596e396405f9997547cc648f47ea640e2404 WatchSource:0}: Error finding container 524f8c2133c54d5b95bcdd938af3596e396405f9997547cc648f47ea640e2404: Status 404 returned error can't find the container with id 524f8c2133c54d5b95bcdd938af3596e396405f9997547cc648f47ea640e2404
Nov 03 21:30:50.380639 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:50.380552    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eeb46f-qr5gs" event={"ID":"99a9cf09-1a48-4a77-9127-9ceb9279708e","Type":"ContainerStarted","Data":"5233a4f6be8f6022c92e12ff14e181f14cc9661dcfd371def2db5cf95d6a2148"}
Nov 03 21:30:50.380639 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:50.380631    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eeb46f-qr5gs" event={"ID":"99a9cf09-1a48-4a77-9127-9ceb9279708e","Type":"ContainerStarted","Data":"524f8c2133c54d5b95bcdd938af3596e396405f9997547cc648f47ea640e2404"}
Nov 03 21:30:50.499575 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:50.499522    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eeb46f-qr5gs" podStartSLOduration=1.499506188 podStartE2EDuration="1.499506188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:50.496814604 +0000 UTC m=+964.990736478" watchObservedRunningTime="2026-11-03 21:30:50.499506188 +0000 UTC m=+964.993428096"
Nov 03 21:30:51.389852 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:51.389811    5053 generic.go:356] "Generic (PLEG): container finished" podID="99a9cf09-1a48-4a77-9127-9ceb9279708e" containerID="5233a4f6be8f6022c92e12ff14e181f14cc9661dcfd371def2db5cf95d6a2148" exitCode=0
Nov 03 21:30:51.390740 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:51.390717    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eeb46f-qr5gs" event={"ID":"99a9cf09-1a48-4a77-9127-9ceb9279708e","Type":"ContainerDied","Data":"5233a4f6be8f6022c92e12ff14e181f14cc9661dcfd371def2db5cf95d6a2148"}
Nov 03 21:30:52.728472 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:52.728366    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eeb46f-qr5gs"
Nov 03 21:30:52.789412 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:52.788751    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99a9cf09-1a48-4a77-9127-9ceb9279708e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a9cf09-1a48-4a77-9127-9ceb9279708e-maas-api-openshift-ca\") pod \"99a9cf09-1a48-4a77-9127-9ceb9279708e\" (UID: \"99a9cf09-1a48-4a77-9127-9ceb9279708e\") "
Nov 03 21:30:52.789412 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:52.788846    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99a9cf09-1a48-4a77-9127-9ceb9279708e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a9cf09-1a48-4a77-9127-9ceb9279708e-maas-api-xks-ca\") pod \"99a9cf09-1a48-4a77-9127-9ceb9279708e\" (UID: \"99a9cf09-1a48-4a77-9127-9ceb9279708e\") "
Nov 03 21:30:52.789412 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:52.788874    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a9cf09-1a48-4a77-9127-9ceb9279708e-kube-api-access-dwc5v\" (UniqueName: \"kubernetes.io/projected/99a9cf09-1a48-4a77-9127-9ceb9279708e-kube-api-access-dwc5v\") pod \"99a9cf09-1a48-4a77-9127-9ceb9279708e\" (UID: \"99a9cf09-1a48-4a77-9127-9ceb9279708e\") "
Nov 03 21:30:52.789722 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:52.789540    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a9cf09-1a48-4a77-9127-9ceb9279708e-maas-api-openshift-ca" pod "99a9cf09-1a48-4a77-9127-9ceb9279708e" (UID: "99a9cf09-1a48-4a77-9127-9ceb9279708e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:30:52.792195 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:52.792134    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99a9cf09-1a48-4a77-9127-9ceb9279708e-maas-api-xks-ca" pod "99a9cf09-1a48-4a77-9127-9ceb9279708e" (UID: "99a9cf09-1a48-4a77-9127-9ceb9279708e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:30:52.793271 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:52.792415    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a9cf09-1a48-4a77-9127-9ceb9279708e-kube-api-access-dwc5v" pod "99a9cf09-1a48-4a77-9127-9ceb9279708e" (UID: "99a9cf09-1a48-4a77-9127-9ceb9279708e"). InnerVolumeSpecName "kube-api-access-dwc5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:30:52.890680 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:52.890646    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a9cf09-1a48-4a77-9127-9ceb9279708e-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:52.890680 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:52.890678    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a9cf09-1a48-4a77-9127-9ceb9279708e-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:52.890949 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:52.890693    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwc5v\" (UniqueName: \"kubernetes.io/projected/99a9cf09-1a48-4a77-9127-9ceb9279708e-kube-api-access-dwc5v\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:53.343260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:53.342577    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8jgdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:30:53.343260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:53.342693    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx" podUID="909e67ce-5511-4f93-91eb-17f6a87acd84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:30:53.430263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:53.429612    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eeb46f-qr5gs" event={"ID":"99a9cf09-1a48-4a77-9127-9ceb9279708e","Type":"ContainerDied","Data":"524f8c2133c54d5b95bcdd938af3596e396405f9997547cc648f47ea640e2404"}
Nov 03 21:30:53.430263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:53.429715    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="524f8c2133c54d5b95bcdd938af3596e396405f9997547cc648f47ea640e2404"
Nov 03 21:30:53.430263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:53.429832    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eeb46f-qr5gs"
Nov 03 21:30:58.368344 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:30:58.368311    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5"
Nov 03 21:31:10.927109 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:31:10.927071    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:31:11.409603 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:11.409568    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6666f84558-gxqpf"]
Nov 03 21:31:11.410169 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:11.410152    5053 cpu_manager.go:414] "containerMap: removing container" podUID="99a9cf09-1a48-4a77-9127-9ceb9279708e" containerName="revoke-keys"
Nov 03 21:31:11.410169 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:11.410170    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a9cf09-1a48-4a77-9127-9ceb9279708e" containerName="revoke-keys"
Nov 03 21:31:11.410326 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:11.410301    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a9cf09-1a48-4a77-9127-9ceb9279708e" containerName="revoke-keys"
Nov 03 21:31:11.413807 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:11.413792    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf"
Nov 03 21:31:11.470414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:11.470375    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6666f84558-gxqpf"]
Nov 03 21:31:11.523931 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:11.523887    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9179a8f2-dfab-4d37-a197-fa9313bb7489-routing-overlay-volume\") pod \"payload-processing-external-model-6666f84558-gxqpf\" (UID: \"9179a8f2-dfab-4d37-a197-fa9313bb7489\") " pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf"
Nov 03 21:31:11.524121 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:11.524064    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9179a8f2-dfab-4d37-a197-fa9313bb7489-plugins-config-volume\") pod \"payload-processing-external-model-6666f84558-gxqpf\" (UID: \"9179a8f2-dfab-4d37-a197-fa9313bb7489\") " pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf"
Nov 03 21:31:11.524121 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:11.524096    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9179a8f2-dfab-4d37-a197-fa9313bb7489-provider-credentials\") pod \"payload-processing-external-model-6666f84558-gxqpf\" (UID: \"9179a8f2-dfab-4d37-a197-fa9313bb7489\") " pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf"
Nov 03 21:31:11.524249 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:11.524163    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llb2d\" (UniqueName: \"kubernetes.io/projected/9179a8f2-dfab-4d37-a197-fa9313bb7489-kube-api-access-llb2d\") pod \"payload-processing-external-model-6666f84558-gxqpf\" (UID: \"9179a8f2-dfab-4d37-a197-fa9313bb7489\") " pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf"
Nov 03 21:31:11.625841 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:11.625809    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9179a8f2-dfab-4d37-a197-fa9313bb7489-plugins-config-volume\") pod \"payload-processing-external-model-6666f84558-gxqpf\" (UID: \"9179a8f2-dfab-4d37-a197-fa9313bb7489\") " pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf"
Nov 03 21:31:11.625841 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:11.625842    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9179a8f2-dfab-4d37-a197-fa9313bb7489-provider-credentials\") pod \"payload-processing-external-model-6666f84558-gxqpf\" (UID: \"9179a8f2-dfab-4d37-a197-fa9313bb7489\") " pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf"
Nov 03 21:31:11.626107 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:11.625871    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llb2d\" (UniqueName: \"kubernetes.io/projected/9179a8f2-dfab-4d37-a197-fa9313bb7489-kube-api-access-llb2d\") pod \"payload-processing-external-model-6666f84558-gxqpf\" (UID: \"9179a8f2-dfab-4d37-a197-fa9313bb7489\") " pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf"
Nov 03 21:31:11.626107 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:11.625937    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9179a8f2-dfab-4d37-a197-fa9313bb7489-routing-overlay-volume\") pod \"payload-processing-external-model-6666f84558-gxqpf\" (UID: \"9179a8f2-dfab-4d37-a197-fa9313bb7489\") " pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf"
Nov 03 21:31:11.626609 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:11.626581    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9179a8f2-dfab-4d37-a197-fa9313bb7489-routing-overlay-volume\") pod \"payload-processing-external-model-6666f84558-gxqpf\" (UID: \"9179a8f2-dfab-4d37-a197-fa9313bb7489\") " pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf"
Nov 03 21:31:11.626748 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:11.626644    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9179a8f2-dfab-4d37-a197-fa9313bb7489-plugins-config-volume\") pod \"payload-processing-external-model-6666f84558-gxqpf\" (UID: \"9179a8f2-dfab-4d37-a197-fa9313bb7489\") " pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf"
Nov 03 21:31:11.628491 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:11.628470    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9179a8f2-dfab-4d37-a197-fa9313bb7489-provider-credentials\") pod \"payload-processing-external-model-6666f84558-gxqpf\" (UID: \"9179a8f2-dfab-4d37-a197-fa9313bb7489\") " pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf"
Nov 03 21:31:11.682071 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:11.682001    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llb2d\" (UniqueName: \"kubernetes.io/projected/9179a8f2-dfab-4d37-a197-fa9313bb7489-kube-api-access-llb2d\") pod \"payload-processing-external-model-6666f84558-gxqpf\" (UID: \"9179a8f2-dfab-4d37-a197-fa9313bb7489\") " pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf"
Nov 03 21:31:11.727225 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:11.727191    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf"
Nov 03 21:31:12.102197 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:12.102162    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6666f84558-gxqpf"]
Nov 03 21:31:12.103978 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:31:12.103950    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9179a8f2_dfab_4d37_a197_fa9313bb7489.slice/crio-71b891d19cc0fdeff9de7739b959861871d59bc0552df8753ff1a1835ae46486 WatchSource:0}: Error finding container 71b891d19cc0fdeff9de7739b959861871d59bc0552df8753ff1a1835ae46486: Status 404 returned error can't find the container with id 71b891d19cc0fdeff9de7739b959861871d59bc0552df8753ff1a1835ae46486
Nov 03 21:31:12.594601 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:12.594557    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf" event={"ID":"9179a8f2-dfab-4d37-a197-fa9313bb7489","Type":"ContainerStarted","Data":"c4a99f10b3d939a9233b1a8aa30038a2272ebbea68b3bfa3f1f57a9022f3aa33"}
Nov 03 21:31:12.594768 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:12.594628    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf" event={"ID":"9179a8f2-dfab-4d37-a197-fa9313bb7489","Type":"ContainerStarted","Data":"71b891d19cc0fdeff9de7739b959861871d59bc0552df8753ff1a1835ae46486"}
Nov 03 21:31:12.594768 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:12.594745    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf"
Nov 03 21:31:12.709311 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:12.709230    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf" podStartSLOduration=1.709208953 podStartE2EDuration="1.709208953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:12.706932088 +0000 UTC m=+987.200853962" watchObservedRunningTime="2026-11-03 21:31:12.709208953 +0000 UTC m=+987.203130827"
Nov 03 21:31:16.512869 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.512833    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"]
Nov 03 21:31:16.517207 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.517185    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.581526 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.581488    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.581526 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.581531    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.581717 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.581559    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.581717 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.581677    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.581717 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.581714    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.581812 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.581737    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v86d\" (UniqueName: \"kubernetes.io/projected/469926c5-1bd5-415b-9c6a-54f4ff10f25a-kube-api-access-8v86d\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.581844 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.581802    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.581894 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.581879    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.581928 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.581903    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.683257 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.683191    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.683453 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.683273    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.683453 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.683322    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.683453 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.683412    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.683612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.683454    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.683612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.683489    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v86d\" (UniqueName: \"kubernetes.io/projected/469926c5-1bd5-415b-9c6a-54f4ff10f25a-kube-api-access-8v86d\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.683710 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.683676    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.683824 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.683798    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.683910 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.683525    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.683963 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.683928    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.683963 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.683958    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.684184 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.684155    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.684396 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.684375    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.684850 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.684824    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.686892 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.686869    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.687403 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.687382    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.719602 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.719571    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"]
Nov 03 21:31:16.741383 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.741356    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v86d\" (UniqueName: \"kubernetes.io/projected/469926c5-1bd5-415b-9c6a-54f4ff10f25a-kube-api-access-8v86d\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.741796 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.741772    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-jlbj5\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:16.830762 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:16.830671    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:17.256543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:17.256517    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"]
Nov 03 21:31:17.257443 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:31:17.257420    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469926c5_1bd5_415b_9c6a_54f4ff10f25a.slice/crio-cb5ebeb3d84f8799d7f2fe6e63f0000fb2c96cf0461f661413a6c9936d4a0b04 WatchSource:0}: Error finding container cb5ebeb3d84f8799d7f2fe6e63f0000fb2c96cf0461f661413a6c9936d4a0b04: Status 404 returned error can't find the container with id cb5ebeb3d84f8799d7f2fe6e63f0000fb2c96cf0461f661413a6c9936d4a0b04
Nov 03 21:31:17.259657 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:17.259623    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:31:17.259738 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:17.259695    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:31:17.259738 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:17.259728    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:31:17.646257 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:17.643710    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5" event={"ID":"469926c5-1bd5-415b-9c6a-54f4ff10f25a","Type":"ContainerStarted","Data":"d19f8a9ae0153d34cadf3c27e26b5afc6fd37bcad4c5665a98f3cb68a7cd4bb1"}
Nov 03 21:31:17.646257 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:17.643778    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5" event={"ID":"469926c5-1bd5-415b-9c6a-54f4ff10f25a","Type":"ContainerStarted","Data":"cb5ebeb3d84f8799d7f2fe6e63f0000fb2c96cf0461f661413a6c9936d4a0b04"}
Nov 03 21:31:17.831736 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:17.831706    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:17.860472 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:17.860425    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5" podStartSLOduration=1.8604113500000001 podStartE2EDuration="1.86041135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:17.810526139 +0000 UTC m=+992.304448028" watchObservedRunningTime="2026-11-03 21:31:17.86041135 +0000 UTC m=+992.354333225"
Nov 03 21:31:18.835877 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:18.835843    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-jlbj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 21:31:18.836394 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:18.836323    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5" podUID="469926c5-1bd5-415b-9c6a-54f4ff10f25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 21:31:19.831751 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:19.831716    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-jlbj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 21:31:19.831911 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:19.831770    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5" podUID="469926c5-1bd5-415b-9c6a-54f4ff10f25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 21:31:20.833261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:20.832666    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-jlbj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 21:31:20.833261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:20.832731    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5" podUID="469926c5-1bd5-415b-9c6a-54f4ff10f25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 21:31:21.835263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:21.835199    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-jlbj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 21:31:21.835789 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:21.835287    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5" podUID="469926c5-1bd5-415b-9c6a-54f4ff10f25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 21:31:22.836263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:22.835549    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-jlbj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 21:31:22.836263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:22.835609    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5" podUID="469926c5-1bd5-415b-9c6a-54f4ff10f25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 21:31:23.338543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:23.337435    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8jgdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:31:23.338543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:23.337529    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx" podUID="909e67ce-5511-4f93-91eb-17f6a87acd84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:31:23.604738 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:23.604652    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf" podUID="9179a8f2-dfab-4d37-a197-fa9313bb7489" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:31:23.831770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:23.831735    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-jlbj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 21:31:23.831981 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:23.831806    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5" podUID="469926c5-1bd5-415b-9c6a-54f4ff10f25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 21:31:24.831414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:24.831378    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-jlbj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 21:31:24.831792 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:24.831453    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5" podUID="469926c5-1bd5-415b-9c6a-54f4ff10f25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 21:31:26.872426 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:26.872385    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-jlbj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:31:26.872891 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:26.872462    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5" podUID="469926c5-1bd5-415b-9c6a-54f4ff10f25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:31:27.774144 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:27.774105    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:27.774463 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:27.774440    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:27.800694 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:27.800663    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:31:33.602560 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:33.602519    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf" podUID="9179a8f2-dfab-4d37-a197-fa9313bb7489" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:31:40.929732 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:31:40.929694    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:31:43.602931 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:43.602890    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf" podUID="9179a8f2-dfab-4d37-a197-fa9313bb7489" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:31:44.185225 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:44.185183    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"]
Nov 03 21:31:44.189472 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:44.189455    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:31:44.247197 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:44.247166    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"]
Nov 03 21:31:44.292813 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:44.292776    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6de8e6\"/\"distinct-w4-6de8e6-kserve-self-signed-certs\""
Nov 03 21:31:44.348222 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:44.348183    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-dshm\") pod \"distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8\" (UID: \"43404f1c-c87c-4325-ab97-bffb6ea81e57\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:31:44.348468 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:44.348381    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-tmp-dir\") pod \"distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8\" (UID: \"43404f1c-c87c-4325-ab97-bffb6ea81e57\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:31:44.348468 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:44.348429    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-model-cache\") pod \"distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8\" (UID: \"43404f1c-c87c-4325-ab97-bffb6ea81e57\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:31:44.348468 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:44.348445    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43404f1c-c87c-4325-ab97-bffb6ea81e57-tls-certs\") pod \"distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8\" (UID: \"43404f1c-c87c-4325-ab97-bffb6ea81e57\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:31:44.348652 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:44.348547    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-home\") pod \"distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8\" (UID: \"43404f1c-c87c-4325-ab97-bffb6ea81e57\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:31:44.348652 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:44.348594    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rwdf\" (UniqueName: \"kubernetes.io/projected/43404f1c-c87c-4325-ab97-bffb6ea81e57-kube-api-access-7rwdf\") pod \"distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8\" (UID: \"43404f1c-c87c-4325-ab97-bffb6ea81e57\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:31:44.450166 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:44.450132    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-tmp-dir\") pod \"distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8\" (UID: \"43404f1c-c87c-4325-ab97-bffb6ea81e57\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:31:44.450381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:44.450182    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-model-cache\") pod \"distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8\" (UID: \"43404f1c-c87c-4325-ab97-bffb6ea81e57\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:31:44.450381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:44.450201    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43404f1c-c87c-4325-ab97-bffb6ea81e57-tls-certs\") pod \"distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8\" (UID: \"43404f1c-c87c-4325-ab97-bffb6ea81e57\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:31:44.450381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:44.450223    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-home\") pod \"distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8\" (UID: \"43404f1c-c87c-4325-ab97-bffb6ea81e57\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:31:44.450381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:44.450278    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rwdf\" (UniqueName: \"kubernetes.io/projected/43404f1c-c87c-4325-ab97-bffb6ea81e57-kube-api-access-7rwdf\") pod \"distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8\" (UID: \"43404f1c-c87c-4325-ab97-bffb6ea81e57\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:31:44.450381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:44.450333    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-dshm\") pod \"distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8\" (UID: \"43404f1c-c87c-4325-ab97-bffb6ea81e57\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:31:44.450653 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:44.450575    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-model-cache\") pod \"distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8\" (UID: \"43404f1c-c87c-4325-ab97-bffb6ea81e57\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:31:44.450653 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:44.450625    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-tmp-dir\") pod \"distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8\" (UID: \"43404f1c-c87c-4325-ab97-bffb6ea81e57\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:31:44.450753 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:44.450688    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-home\") pod \"distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8\" (UID: \"43404f1c-c87c-4325-ab97-bffb6ea81e57\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:31:44.452812 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:44.452778    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-dshm\") pod \"distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8\" (UID: \"43404f1c-c87c-4325-ab97-bffb6ea81e57\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:31:44.453057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:44.453039    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43404f1c-c87c-4325-ab97-bffb6ea81e57-tls-certs\") pod \"distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8\" (UID: \"43404f1c-c87c-4325-ab97-bffb6ea81e57\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:31:44.507294 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:44.507261    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rwdf\" (UniqueName: \"kubernetes.io/projected/43404f1c-c87c-4325-ab97-bffb6ea81e57-kube-api-access-7rwdf\") pod \"distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8\" (UID: \"43404f1c-c87c-4325-ab97-bffb6ea81e57\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:31:44.802941 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:44.802846    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:31:45.179542 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:45.179510    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"]
Nov 03 21:31:45.181022 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:31:45.180997    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43404f1c_c87c_4325_ab97_bffb6ea81e57.slice/crio-d4e6a5ad2e1e435f0b2e28bdc0219228fd2a48e1599e47a6b6b15abd3746e750 WatchSource:0}: Error finding container d4e6a5ad2e1e435f0b2e28bdc0219228fd2a48e1599e47a6b6b15abd3746e750: Status 404 returned error can't find the container with id d4e6a5ad2e1e435f0b2e28bdc0219228fd2a48e1599e47a6b6b15abd3746e750
Nov 03 21:31:45.859027 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:45.858989    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8" event={"ID":"43404f1c-c87c-4325-ab97-bffb6ea81e57","Type":"ContainerStarted","Data":"05761c88f26f13861c837676cdcec7871393c6846daddbe2d7fa5993e9b8a78e"}
Nov 03 21:31:45.859478 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:45.859037    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8" event={"ID":"43404f1c-c87c-4325-ab97-bffb6ea81e57","Type":"ContainerStarted","Data":"d4e6a5ad2e1e435f0b2e28bdc0219228fd2a48e1599e47a6b6b15abd3746e750"}
Nov 03 21:31:45.900556 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:45.900527    5053 scope.go:122] "RemoveContainer" containerID="4d83383e87d16da6ae10c0ff92ebcc63e7ecdbd187657a7176403ec40abe256f"
Nov 03 21:31:45.974084 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:45.974028    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8" podStartSLOduration=1.974014274 podStartE2EDuration="1.974014274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:45.97102416 +0000 UTC m=+1020.464946038" watchObservedRunningTime="2026-11-03 21:31:45.974014274 +0000 UTC m=+1020.467936149"
Nov 03 21:31:46.803972 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:46.803939    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:31:53.602850 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:53.602806    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf" podUID="9179a8f2-dfab-4d37-a197-fa9313bb7489" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:31:54.803519 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:54.803461    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:31:54.817185 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:54.817156    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:31:54.939349 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:31:54.939316    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:32:03.602740 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:03.602695    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf" podUID="9179a8f2-dfab-4d37-a197-fa9313bb7489" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:32:10.003700 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:10.003664    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6666f84558-gxqpf"]
Nov 03 21:32:10.005011 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:10.004970    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf" podUID="9179a8f2-dfab-4d37-a197-fa9313bb7489" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:32:10.028632 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:10.028577    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6575949dfc-55wmh"]
Nov 03 21:32:10.033311 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:10.033281    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6575949dfc-55wmh"
Nov 03 21:32:10.116779 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:10.116744    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6575949dfc-55wmh"]
Nov 03 21:32:10.165969 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:10.165920    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-provider-credentials\") pod \"payload-processing-external-model-6575949dfc-55wmh\" (UID: \"bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296\") " pod="models-as-a-service/payload-processing-external-model-6575949dfc-55wmh"
Nov 03 21:32:10.166154 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:10.166119    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k48p\" (UniqueName: \"kubernetes.io/projected/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-kube-api-access-6k48p\") pod \"payload-processing-external-model-6575949dfc-55wmh\" (UID: \"bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296\") " pod="models-as-a-service/payload-processing-external-model-6575949dfc-55wmh"
Nov 03 21:32:10.166200 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:10.166175    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-routing-overlay-volume\") pod \"payload-processing-external-model-6575949dfc-55wmh\" (UID: \"bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296\") " pod="models-as-a-service/payload-processing-external-model-6575949dfc-55wmh"
Nov 03 21:32:10.166430 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:10.166408    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-plugins-config-volume\") pod \"payload-processing-external-model-6575949dfc-55wmh\" (UID: \"bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296\") " pod="models-as-a-service/payload-processing-external-model-6575949dfc-55wmh"
Nov 03 21:32:10.267522 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:10.267418    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k48p\" (UniqueName: \"kubernetes.io/projected/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-kube-api-access-6k48p\") pod \"payload-processing-external-model-6575949dfc-55wmh\" (UID: \"bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296\") " pod="models-as-a-service/payload-processing-external-model-6575949dfc-55wmh"
Nov 03 21:32:10.267522 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:10.267504    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-routing-overlay-volume\") pod \"payload-processing-external-model-6575949dfc-55wmh\" (UID: \"bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296\") " pod="models-as-a-service/payload-processing-external-model-6575949dfc-55wmh"
Nov 03 21:32:10.267780 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:10.267628    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-plugins-config-volume\") pod \"payload-processing-external-model-6575949dfc-55wmh\" (UID: \"bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296\") " pod="models-as-a-service/payload-processing-external-model-6575949dfc-55wmh"
Nov 03 21:32:10.267780 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:10.267677    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-provider-credentials\") pod \"payload-processing-external-model-6575949dfc-55wmh\" (UID: \"bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296\") " pod="models-as-a-service/payload-processing-external-model-6575949dfc-55wmh"
Nov 03 21:32:10.268215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:10.268191    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-routing-overlay-volume\") pod \"payload-processing-external-model-6575949dfc-55wmh\" (UID: \"bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296\") " pod="models-as-a-service/payload-processing-external-model-6575949dfc-55wmh"
Nov 03 21:32:10.268359 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:10.268339    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-plugins-config-volume\") pod \"payload-processing-external-model-6575949dfc-55wmh\" (UID: \"bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296\") " pod="models-as-a-service/payload-processing-external-model-6575949dfc-55wmh"
Nov 03 21:32:10.270355 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:10.270335    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-provider-credentials\") pod \"payload-processing-external-model-6575949dfc-55wmh\" (UID: \"bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296\") " pod="models-as-a-service/payload-processing-external-model-6575949dfc-55wmh"
Nov 03 21:32:10.324861 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:10.324828    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k48p\" (UniqueName: \"kubernetes.io/projected/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-kube-api-access-6k48p\") pod \"payload-processing-external-model-6575949dfc-55wmh\" (UID: \"bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296\") " pod="models-as-a-service/payload-processing-external-model-6575949dfc-55wmh"
Nov 03 21:32:10.347794 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:10.347756    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6575949dfc-55wmh"
Nov 03 21:32:10.728632 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:10.728597    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6575949dfc-55wmh"]
Nov 03 21:32:10.728818 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:32:10.728792    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbc0f4d_eb3b_4057_9ae6_9b6ce6ce3296.slice/crio-2cf4a230238b88db3886024d693d36d3739a16f41c811213d0531926b9fe56fa WatchSource:0}: Error finding container 2cf4a230238b88db3886024d693d36d3739a16f41c811213d0531926b9fe56fa: Status 404 returned error can't find the container with id 2cf4a230238b88db3886024d693d36d3739a16f41c811213d0531926b9fe56fa
Nov 03 21:32:10.936189 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:32:10.936146    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:32:11.045492 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:11.045403    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6575949dfc-55wmh" event={"ID":"bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296","Type":"ContainerStarted","Data":"8ac5b01b14d625cec79d9287ecf067e5af8f04a1dd00d706ab67fe200a10da05"}
Nov 03 21:32:11.045492 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:11.045462    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6575949dfc-55wmh" event={"ID":"bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296","Type":"ContainerStarted","Data":"2cf4a230238b88db3886024d693d36d3739a16f41c811213d0531926b9fe56fa"}
Nov 03 21:32:11.045903 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:11.045509    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6575949dfc-55wmh"
Nov 03 21:32:11.159347 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:11.159283    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6575949dfc-55wmh" podStartSLOduration=2.159264617 podStartE2EDuration="2.159264617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:32:11.15760145 +0000 UTC m=+1045.651523337" watchObservedRunningTime="2026-11-03 21:32:11.159264617 +0000 UTC m=+1045.653186494"
Nov 03 21:32:15.008264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:15.005328    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf" podUID="9179a8f2-dfab-4d37-a197-fa9313bb7489" containerName="payload-processing" containerID="cri-o://c4a99f10b3d939a9233b1a8aa30038a2272ebbea68b3bfa3f1f57a9022f3aa33" gracePeriod=25
Nov 03 21:32:15.756296 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:15.756016    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf"
Nov 03 21:32:15.849355 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:15.849226    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9179a8f2-dfab-4d37-a197-fa9313bb7489-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9179a8f2-dfab-4d37-a197-fa9313bb7489-provider-credentials\") pod \"9179a8f2-dfab-4d37-a197-fa9313bb7489\" (UID: \"9179a8f2-dfab-4d37-a197-fa9313bb7489\") "
Nov 03 21:32:15.849581 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:15.849567    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9179a8f2-dfab-4d37-a197-fa9313bb7489-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9179a8f2-dfab-4d37-a197-fa9313bb7489-routing-overlay-volume\") pod \"9179a8f2-dfab-4d37-a197-fa9313bb7489\" (UID: \"9179a8f2-dfab-4d37-a197-fa9313bb7489\") "
Nov 03 21:32:15.849666 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:15.849656    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9179a8f2-dfab-4d37-a197-fa9313bb7489-kube-api-access-llb2d\" (UniqueName: \"kubernetes.io/projected/9179a8f2-dfab-4d37-a197-fa9313bb7489-kube-api-access-llb2d\") pod \"9179a8f2-dfab-4d37-a197-fa9313bb7489\" (UID: \"9179a8f2-dfab-4d37-a197-fa9313bb7489\") "
Nov 03 21:32:15.849752 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:15.849742    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9179a8f2-dfab-4d37-a197-fa9313bb7489-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9179a8f2-dfab-4d37-a197-fa9313bb7489-plugins-config-volume\") pod \"9179a8f2-dfab-4d37-a197-fa9313bb7489\" (UID: \"9179a8f2-dfab-4d37-a197-fa9313bb7489\") "
Nov 03 21:32:15.850540 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:15.850519    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9179a8f2-dfab-4d37-a197-fa9313bb7489-plugins-config-volume" pod "9179a8f2-dfab-4d37-a197-fa9313bb7489" (UID: "9179a8f2-dfab-4d37-a197-fa9313bb7489"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:32:15.851393 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:15.851366    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9179a8f2-dfab-4d37-a197-fa9313bb7489-routing-overlay-volume" pod "9179a8f2-dfab-4d37-a197-fa9313bb7489" (UID: "9179a8f2-dfab-4d37-a197-fa9313bb7489"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:32:15.858837 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:15.858790    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9179a8f2-dfab-4d37-a197-fa9313bb7489-provider-credentials" pod "9179a8f2-dfab-4d37-a197-fa9313bb7489" (UID: "9179a8f2-dfab-4d37-a197-fa9313bb7489"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:32:15.860879 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:15.860838    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9179a8f2-dfab-4d37-a197-fa9313bb7489-kube-api-access-llb2d" pod "9179a8f2-dfab-4d37-a197-fa9313bb7489" (UID: "9179a8f2-dfab-4d37-a197-fa9313bb7489"). InnerVolumeSpecName "kube-api-access-llb2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:32:15.951281 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:15.951249    5053 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9179a8f2-dfab-4d37-a197-fa9313bb7489-provider-credentials\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:32:15.951281 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:15.951277    5053 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9179a8f2-dfab-4d37-a197-fa9313bb7489-routing-overlay-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:32:15.951281 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:15.951287    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llb2d\" (UniqueName: \"kubernetes.io/projected/9179a8f2-dfab-4d37-a197-fa9313bb7489-kube-api-access-llb2d\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:32:15.951609 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:15.951297    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9179a8f2-dfab-4d37-a197-fa9313bb7489-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:32:16.100076 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:16.099995    5053 generic.go:356] "Generic (PLEG): container finished" podID="9179a8f2-dfab-4d37-a197-fa9313bb7489" containerID="c4a99f10b3d939a9233b1a8aa30038a2272ebbea68b3bfa3f1f57a9022f3aa33" exitCode=0
Nov 03 21:32:16.100076 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:16.100039    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf"
Nov 03 21:32:16.100576 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:16.100070    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf" event={"ID":"9179a8f2-dfab-4d37-a197-fa9313bb7489","Type":"ContainerDied","Data":"c4a99f10b3d939a9233b1a8aa30038a2272ebbea68b3bfa3f1f57a9022f3aa33"}
Nov 03 21:32:16.100576 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:16.100129    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6666f84558-gxqpf" event={"ID":"9179a8f2-dfab-4d37-a197-fa9313bb7489","Type":"ContainerDied","Data":"71b891d19cc0fdeff9de7739b959861871d59bc0552df8753ff1a1835ae46486"}
Nov 03 21:32:16.100576 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:16.100158    5053 scope.go:122] "RemoveContainer" containerID="c4a99f10b3d939a9233b1a8aa30038a2272ebbea68b3bfa3f1f57a9022f3aa33"
Nov 03 21:32:16.115738 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:16.113511    5053 scope.go:122] "RemoveContainer" containerID="c4a99f10b3d939a9233b1a8aa30038a2272ebbea68b3bfa3f1f57a9022f3aa33"
Nov 03 21:32:16.115738 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:32:16.115651    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a99f10b3d939a9233b1a8aa30038a2272ebbea68b3bfa3f1f57a9022f3aa33\": container with ID starting with c4a99f10b3d939a9233b1a8aa30038a2272ebbea68b3bfa3f1f57a9022f3aa33 not found: ID does not exist" containerID="c4a99f10b3d939a9233b1a8aa30038a2272ebbea68b3bfa3f1f57a9022f3aa33"
Nov 03 21:32:16.115738 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:16.115686    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a99f10b3d939a9233b1a8aa30038a2272ebbea68b3bfa3f1f57a9022f3aa33"} err="failed to get container status \"c4a99f10b3d939a9233b1a8aa30038a2272ebbea68b3bfa3f1f57a9022f3aa33\": rpc error: code = NotFound desc = could not find container \"c4a99f10b3d939a9233b1a8aa30038a2272ebbea68b3bfa3f1f57a9022f3aa33\": container with ID starting with c4a99f10b3d939a9233b1a8aa30038a2272ebbea68b3bfa3f1f57a9022f3aa33 not found: ID does not exist"
Nov 03 21:32:16.266734 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:16.266685    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6666f84558-gxqpf"]
Nov 03 21:32:16.271472 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:16.271447    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6666f84558-gxqpf"]
Nov 03 21:32:17.667101 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:17.667068    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9179a8f2-dfab-4d37-a197-fa9313bb7489" path="/var/lib/kubelet/pods/9179a8f2-dfab-4d37-a197-fa9313bb7489/volumes"
Nov 03 21:32:18.010509 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.010478    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"]
Nov 03 21:32:18.011084 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.011068    5053 cpu_manager.go:414] "containerMap: removing container" podUID="9179a8f2-dfab-4d37-a197-fa9313bb7489" containerName="payload-processing"
Nov 03 21:32:18.011160 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.011086    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9179a8f2-dfab-4d37-a197-fa9313bb7489" containerName="payload-processing"
Nov 03 21:32:18.011205 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.011192    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="9179a8f2-dfab-4d37-a197-fa9313bb7489" containerName="payload-processing"
Nov 03 21:32:18.016151 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.016134    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.077096 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.077056    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.077096 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.077098    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.077425 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.077298    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zvvf\" (UniqueName: \"kubernetes.io/projected/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-kube-api-access-2zvvf\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.077425 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.077346    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.077497 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.077443    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.077544 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.077502    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.077544 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.077521    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.077630 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.077616    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.077676 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.077665    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.144103 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.144067    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"]
Nov 03 21:32:18.178467 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.178435    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.178668 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.178488    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.178668 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.178535    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.178668 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.178566    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.178668 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.178660    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zvvf\" (UniqueName: \"kubernetes.io/projected/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-kube-api-access-2zvvf\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.178886 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.178694    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.178886 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.178748    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.178886 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.178814    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.178886 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.178837    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.179071 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.178896    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.179071 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.178980    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.179179 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.179159    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.179254 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.179211    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.179625 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.179601    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.181934 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.181911    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.182260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.182222    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.235195 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.235158    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zvvf\" (UniqueName: \"kubernetes.io/projected/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-kube-api-access-2zvvf\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.235195 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.235157    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.329946 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.329865    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:18.759830 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.759803    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"]
Nov 03 21:32:18.761542 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:32:18.761514    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a77c28_78e6_42ce_89cd_e9956e13bfc2.slice/crio-ac23ccbe69a838105464b082ab06bf8b0f266aa751597fd62c7406b35e8170b6 WatchSource:0}: Error finding container ac23ccbe69a838105464b082ab06bf8b0f266aa751597fd62c7406b35e8170b6: Status 404 returned error can't find the container with id ac23ccbe69a838105464b082ab06bf8b0f266aa751597fd62c7406b35e8170b6
Nov 03 21:32:18.763821 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.763791    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:32:18.763878 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.763866    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:32:18.763914 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.763893    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:32:18.826140 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.826112    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6575949dfc-55wmh"]
Nov 03 21:32:18.829806 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.829762    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6575949dfc-55wmh" podUID="bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:32:18.856906 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.856868    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7868f59688-cw6fs"]
Nov 03 21:32:18.861189 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.861163    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7868f59688-cw6fs"
Nov 03 21:32:18.888245 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.888191    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f53fe56d-d4f6-4754-a27b-87f71b5895db-plugins-config-volume\") pod \"payload-processing-external-model-7868f59688-cw6fs\" (UID: \"f53fe56d-d4f6-4754-a27b-87f71b5895db\") " pod="models-as-a-service/payload-processing-external-model-7868f59688-cw6fs"
Nov 03 21:32:18.888399 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.888280    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nwps\" (UniqueName: \"kubernetes.io/projected/f53fe56d-d4f6-4754-a27b-87f71b5895db-kube-api-access-4nwps\") pod \"payload-processing-external-model-7868f59688-cw6fs\" (UID: \"f53fe56d-d4f6-4754-a27b-87f71b5895db\") " pod="models-as-a-service/payload-processing-external-model-7868f59688-cw6fs"
Nov 03 21:32:18.888399 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.888377    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f53fe56d-d4f6-4754-a27b-87f71b5895db-provider-credentials\") pod \"payload-processing-external-model-7868f59688-cw6fs\" (UID: \"f53fe56d-d4f6-4754-a27b-87f71b5895db\") " pod="models-as-a-service/payload-processing-external-model-7868f59688-cw6fs"
Nov 03 21:32:18.888492 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.888476    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f53fe56d-d4f6-4754-a27b-87f71b5895db-routing-overlay-volume\") pod \"payload-processing-external-model-7868f59688-cw6fs\" (UID: \"f53fe56d-d4f6-4754-a27b-87f71b5895db\") " pod="models-as-a-service/payload-processing-external-model-7868f59688-cw6fs"
Nov 03 21:32:18.913058 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.912998    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-dd42e4bb025c\""
Nov 03 21:32:18.913058 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.913004    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-518a2975ef10\""
Nov 03 21:32:18.916426 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.916398    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7868f59688-cw6fs"]
Nov 03 21:32:18.990114 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.990077    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f53fe56d-d4f6-4754-a27b-87f71b5895db-routing-overlay-volume\") pod \"payload-processing-external-model-7868f59688-cw6fs\" (UID: \"f53fe56d-d4f6-4754-a27b-87f71b5895db\") " pod="models-as-a-service/payload-processing-external-model-7868f59688-cw6fs"
Nov 03 21:32:18.990341 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.990230    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f53fe56d-d4f6-4754-a27b-87f71b5895db-plugins-config-volume\") pod \"payload-processing-external-model-7868f59688-cw6fs\" (UID: \"f53fe56d-d4f6-4754-a27b-87f71b5895db\") " pod="models-as-a-service/payload-processing-external-model-7868f59688-cw6fs"
Nov 03 21:32:18.990341 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.990306    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nwps\" (UniqueName: \"kubernetes.io/projected/f53fe56d-d4f6-4754-a27b-87f71b5895db-kube-api-access-4nwps\") pod \"payload-processing-external-model-7868f59688-cw6fs\" (UID: \"f53fe56d-d4f6-4754-a27b-87f71b5895db\") " pod="models-as-a-service/payload-processing-external-model-7868f59688-cw6fs"
Nov 03 21:32:18.990478 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.990358    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f53fe56d-d4f6-4754-a27b-87f71b5895db-provider-credentials\") pod \"payload-processing-external-model-7868f59688-cw6fs\" (UID: \"f53fe56d-d4f6-4754-a27b-87f71b5895db\") " pod="models-as-a-service/payload-processing-external-model-7868f59688-cw6fs"
Nov 03 21:32:18.990949 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.990923    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f53fe56d-d4f6-4754-a27b-87f71b5895db-routing-overlay-volume\") pod \"payload-processing-external-model-7868f59688-cw6fs\" (UID: \"f53fe56d-d4f6-4754-a27b-87f71b5895db\") " pod="models-as-a-service/payload-processing-external-model-7868f59688-cw6fs"
Nov 03 21:32:18.991191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.991168    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f53fe56d-d4f6-4754-a27b-87f71b5895db-plugins-config-volume\") pod \"payload-processing-external-model-7868f59688-cw6fs\" (UID: \"f53fe56d-d4f6-4754-a27b-87f71b5895db\") " pod="models-as-a-service/payload-processing-external-model-7868f59688-cw6fs"
Nov 03 21:32:18.993755 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:18.993730    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f53fe56d-d4f6-4754-a27b-87f71b5895db-provider-credentials\") pod \"payload-processing-external-model-7868f59688-cw6fs\" (UID: \"f53fe56d-d4f6-4754-a27b-87f71b5895db\") " pod="models-as-a-service/payload-processing-external-model-7868f59688-cw6fs"
Nov 03 21:32:19.046765 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:19.046681    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nwps\" (UniqueName: \"kubernetes.io/projected/f53fe56d-d4f6-4754-a27b-87f71b5895db-kube-api-access-4nwps\") pod \"payload-processing-external-model-7868f59688-cw6fs\" (UID: \"f53fe56d-d4f6-4754-a27b-87f71b5895db\") " pod="models-as-a-service/payload-processing-external-model-7868f59688-cw6fs"
Nov 03 21:32:19.128988 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:19.128948    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k" event={"ID":"b9a77c28-78e6-42ce-89cd-e9956e13bfc2","Type":"ContainerStarted","Data":"565a74b41c0a9321ac8cdc1edbbadcdde952cfc70d0132c73a76748f5ac113fc"}
Nov 03 21:32:19.129137 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:19.129044    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k" event={"ID":"b9a77c28-78e6-42ce-89cd-e9956e13bfc2","Type":"ContainerStarted","Data":"ac23ccbe69a838105464b082ab06bf8b0f266aa751597fd62c7406b35e8170b6"}
Nov 03 21:32:19.178750 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:19.178714    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7868f59688-cw6fs"
Nov 03 21:32:19.247486 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:19.246555    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k" podStartSLOduration=2.246532432 podStartE2EDuration="2.246532432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:32:19.244272635 +0000 UTC m=+1053.738194508" watchObservedRunningTime="2026-11-03 21:32:19.246532432 +0000 UTC m=+1053.740454365"
Nov 03 21:32:19.330721 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:19.330587    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:19.335033 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:19.334657    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 21:32:19.335033 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:19.334718    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k" podUID="b9a77c28-78e6-42ce-89cd-e9956e13bfc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 21:32:19.558202 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:19.558163    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7868f59688-cw6fs"]
Nov 03 21:32:19.702374 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:32:19.702338    5053 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6bbcc490edc5: secret "agc-cred-6bbcc490edc5" not found
Nov 03 21:32:19.702630 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:32:19.702615    5053 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5: secret "agc-cred-6bbcc490edc5" not found
Nov 03 21:32:19.702793 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:32:19.702780    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-provider-credentials podName:e2736c90-d7e3-4aa1-81dd-45dea43e3b3f nodeName:}" failed. No retries permitted until 2026-11-03 21:32:20.202756947 +0000 UTC m=+1054.696678808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-provider-credentials") pod "payload-processing-external-model-cb5b88ddd-qqqh5" (UID: "e2736c90-d7e3-4aa1-81dd-45dea43e3b3f") : secret "agc-cred-6bbcc490edc5" not found
Nov 03 21:32:20.140276 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:20.140202    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7868f59688-cw6fs" event={"ID":"f53fe56d-d4f6-4754-a27b-87f71b5895db","Type":"ContainerStarted","Data":"fdc801a59da4723f34101bb058bab315ff0a7c4c0c0815d1643ae23ef3482e02"}
Nov 03 21:32:20.140663 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:20.140279    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7868f59688-cw6fs" event={"ID":"f53fe56d-d4f6-4754-a27b-87f71b5895db","Type":"ContainerStarted","Data":"a463220381f4c846ecfcc0d416f4fd967ae08026f289509113bd44d49a18baa0"}
Nov 03 21:32:20.140663 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:20.140310    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7868f59688-cw6fs"
Nov 03 21:32:20.230263 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:32:20.226517    5053 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6bbcc490edc5: secret "agc-cred-6bbcc490edc5" not found
Nov 03 21:32:20.230263 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:32:20.226551    5053 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5: secret "agc-cred-6bbcc490edc5" not found
Nov 03 21:32:20.230263 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:32:20.226608    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-provider-credentials podName:e2736c90-d7e3-4aa1-81dd-45dea43e3b3f nodeName:}" failed. No retries permitted until 2026-11-03 21:32:21.226588759 +0000 UTC m=+1055.720510628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-provider-credentials") pod "payload-processing-external-model-cb5b88ddd-qqqh5" (UID: "e2736c90-d7e3-4aa1-81dd-45dea43e3b3f") : secret "agc-cred-6bbcc490edc5" not found
Nov 03 21:32:20.260040 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:20.259992    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7868f59688-cw6fs" podStartSLOduration=2.2599777899999998 podStartE2EDuration="2.25997779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:32:20.256993047 +0000 UTC m=+1054.750914926" watchObservedRunningTime="2026-11-03 21:32:20.25997779 +0000 UTC m=+1054.753899665"
Nov 03 21:32:20.334782 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:20.334735    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 21:32:20.334988 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:20.334802    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k" podUID="b9a77c28-78e6-42ce-89cd-e9956e13bfc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 21:32:21.245679 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:32:21.245644    5053 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6bbcc490edc5: secret "agc-cred-6bbcc490edc5" not found
Nov 03 21:32:21.246220 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:32:21.246193    5053 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5: secret "agc-cred-6bbcc490edc5" not found
Nov 03 21:32:21.246372 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:32:21.246307    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-provider-credentials podName:e2736c90-d7e3-4aa1-81dd-45dea43e3b3f nodeName:}" failed. No retries permitted until 2026-11-03 21:32:23.246286605 +0000 UTC m=+1057.740208461 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-provider-credentials") pod "payload-processing-external-model-cb5b88ddd-qqqh5" (UID: "e2736c90-d7e3-4aa1-81dd-45dea43e3b3f") : secret "agc-cred-6bbcc490edc5" not found
Nov 03 21:32:21.334081 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:21.333648    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 21:32:21.334081 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:21.333710    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k" podUID="b9a77c28-78e6-42ce-89cd-e9956e13bfc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 21:32:22.334077 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:22.334037    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 21:32:22.334604 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:22.334107    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k" podUID="b9a77c28-78e6-42ce-89cd-e9956e13bfc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 21:32:23.271457 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:32:23.271418    5053 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6bbcc490edc5: secret "agc-cred-6bbcc490edc5" not found
Nov 03 21:32:23.271457 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:32:23.271455    5053 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5: secret "agc-cred-6bbcc490edc5" not found
Nov 03 21:32:23.271671 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:32:23.271510    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-provider-credentials podName:e2736c90-d7e3-4aa1-81dd-45dea43e3b3f nodeName:}" failed. No retries permitted until 2026-11-03 21:32:27.271494046 +0000 UTC m=+1061.765415900 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-provider-credentials") pod "payload-processing-external-model-cb5b88ddd-qqqh5" (UID: "e2736c90-d7e3-4aa1-81dd-45dea43e3b3f") : secret "agc-cred-6bbcc490edc5" not found
Nov 03 21:32:23.332947 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:23.332911    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 21:32:23.333123 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:23.332965    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k" podUID="b9a77c28-78e6-42ce-89cd-e9956e13bfc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 21:32:23.827528 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:23.827308    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6575949dfc-55wmh" podUID="bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296" containerName="payload-processing" containerID="cri-o://8ac5b01b14d625cec79d9287ecf067e5af8f04a1dd00d706ab67fe200a10da05" gracePeriod=25
Nov 03 21:32:24.175174 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:24.175141    5053 generic.go:356] "Generic (PLEG): container finished" podID="bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296" containerID="8ac5b01b14d625cec79d9287ecf067e5af8f04a1dd00d706ab67fe200a10da05" exitCode=0
Nov 03 21:32:24.175364 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:24.175216    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6575949dfc-55wmh" event={"ID":"bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296","Type":"ContainerDied","Data":"8ac5b01b14d625cec79d9287ecf067e5af8f04a1dd00d706ab67fe200a10da05"}
Nov 03 21:32:24.175364 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:24.175317    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6575949dfc-55wmh" event={"ID":"bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296","Type":"ContainerDied","Data":"2cf4a230238b88db3886024d693d36d3739a16f41c811213d0531926b9fe56fa"}
Nov 03 21:32:24.175364 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:24.175335    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cf4a230238b88db3886024d693d36d3739a16f41c811213d0531926b9fe56fa"
Nov 03 21:32:24.177374 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:24.177347    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6575949dfc-55wmh"
Nov 03 21:32:24.284349 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:24.284313    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-provider-credentials\") pod \"bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296\" (UID: \"bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296\") "
Nov 03 21:32:24.284534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:24.284429    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-kube-api-access-6k48p\" (UniqueName: \"kubernetes.io/projected/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-kube-api-access-6k48p\") pod \"bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296\" (UID: \"bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296\") "
Nov 03 21:32:24.284534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:24.284527    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-routing-overlay-volume\") pod \"bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296\" (UID: \"bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296\") "
Nov 03 21:32:24.284733 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:24.284713    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-plugins-config-volume\") pod \"bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296\" (UID: \"bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296\") "
Nov 03 21:32:24.284879 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:24.284854    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-routing-overlay-volume" pod "bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296" (UID: "bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:32:24.285049 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:24.285024    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-plugins-config-volume" pod "bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296" (UID: "bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:32:24.285183 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:24.285168    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:32:24.285257 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:24.285187    5053 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-routing-overlay-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:32:24.287228 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:24.287200    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-kube-api-access-6k48p" pod "bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296" (UID: "bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296"). InnerVolumeSpecName "kube-api-access-6k48p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:32:24.287228 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:24.287210    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-provider-credentials" pod "bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296" (UID: "bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:32:24.336399 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:24.335729    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 21:32:24.336399 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:24.335808    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k" podUID="b9a77c28-78e6-42ce-89cd-e9956e13bfc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 21:32:24.386744 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:24.386626    5053 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-provider-credentials\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:32:24.386744 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:24.386655    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k48p\" (UniqueName: \"kubernetes.io/projected/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296-kube-api-access-6k48p\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:32:25.205563 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:25.205530    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6575949dfc-55wmh"
Nov 03 21:32:25.330769 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:25.330731    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 21:32:25.330986 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:25.330819    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k" podUID="b9a77c28-78e6-42ce-89cd-e9956e13bfc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 21:32:25.380850 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:25.380808    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6575949dfc-55wmh"]
Nov 03 21:32:25.432408 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:25.432359    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6575949dfc-55wmh"]
Nov 03 21:32:25.690332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:25.689330    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296" path="/var/lib/kubelet/pods/bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296/volumes"
Nov 03 21:32:26.337615 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:26.337578    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:26.339025 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:26.339000    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:26.339855 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:26.339832    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:32:27.295347 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:32:27.295307    5053 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6bbcc490edc5: secret "agc-cred-6bbcc490edc5" not found
Nov 03 21:32:27.295347 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:32:27.295344    5053 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5: secret "agc-cred-6bbcc490edc5" not found
Nov 03 21:32:27.295627 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:32:27.295417    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-provider-credentials podName:e2736c90-d7e3-4aa1-81dd-45dea43e3b3f nodeName:}" failed. No retries permitted until 2026-11-03 21:32:35.295396903 +0000 UTC m=+1069.789318762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-provider-credentials") pod "payload-processing-external-model-cb5b88ddd-qqqh5" (UID: "e2736c90-d7e3-4aa1-81dd-45dea43e3b3f") : secret "agc-cred-6bbcc490edc5" not found
Nov 03 21:32:31.152110 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:31.152077    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7868f59688-cw6fs"
Nov 03 21:32:31.319677 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:31.319638    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5"]
Nov 03 21:32:35.337052 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:32:35.337022    5053 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6bbcc490edc5: secret "agc-cred-6bbcc490edc5" not found
Nov 03 21:32:35.337052 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:32:35.337052    5053 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5: secret "agc-cred-6bbcc490edc5" not found
Nov 03 21:32:35.337513 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:32:35.337105    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-provider-credentials podName:e2736c90-d7e3-4aa1-81dd-45dea43e3b3f nodeName:}" failed. No retries permitted until 2026-11-03 21:32:51.337089618 +0000 UTC m=+1085.831011474 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-provider-credentials") pod "payload-processing-external-model-cb5b88ddd-qqqh5" (UID: "e2736c90-d7e3-4aa1-81dd-45dea43e3b3f") : secret "agc-cred-6bbcc490edc5" not found
Nov 03 21:32:36.320578 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:36.320535    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5" podUID="e2736c90-d7e3-4aa1-81dd-45dea43e3b3f" containerName="payload-processing" containerID="cri-o://6bb05c759cb939604368e270cd6f62af0644e7e4d86e5eb613617a729114acb6" gracePeriod=25
Nov 03 21:32:36.583680 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:36.583617    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5"
Nov 03 21:32:36.652444 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:36.652413    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-provider-credentials\") pod \"e2736c90-d7e3-4aa1-81dd-45dea43e3b3f\" (UID: \"e2736c90-d7e3-4aa1-81dd-45dea43e3b3f\") "
Nov 03 21:32:36.652587 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:36.652489    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-plugins-config-volume\") pod \"e2736c90-d7e3-4aa1-81dd-45dea43e3b3f\" (UID: \"e2736c90-d7e3-4aa1-81dd-45dea43e3b3f\") "
Nov 03 21:32:36.652587 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:36.652529    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-routing-overlay-volume\") pod \"e2736c90-d7e3-4aa1-81dd-45dea43e3b3f\" (UID: \"e2736c90-d7e3-4aa1-81dd-45dea43e3b3f\") "
Nov 03 21:32:36.652724 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:36.652592    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-kube-api-access-wbrv8\" (UniqueName: \"kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-kube-api-access-wbrv8\") pod \"e2736c90-d7e3-4aa1-81dd-45dea43e3b3f\" (UID: \"e2736c90-d7e3-4aa1-81dd-45dea43e3b3f\") "
Nov 03 21:32:36.652993 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:36.652958    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-routing-overlay-volume" pod "e2736c90-d7e3-4aa1-81dd-45dea43e3b3f" (UID: "e2736c90-d7e3-4aa1-81dd-45dea43e3b3f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:32:36.653117 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:36.653002    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-plugins-config-volume" pod "e2736c90-d7e3-4aa1-81dd-45dea43e3b3f" (UID: "e2736c90-d7e3-4aa1-81dd-45dea43e3b3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:32:36.653117 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:36.653067    5053 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-routing-overlay-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:32:36.654885 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:36.654864    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-provider-credentials" pod "e2736c90-d7e3-4aa1-81dd-45dea43e3b3f" (UID: "e2736c90-d7e3-4aa1-81dd-45dea43e3b3f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:32:36.654973 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:36.654899    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-kube-api-access-wbrv8" pod "e2736c90-d7e3-4aa1-81dd-45dea43e3b3f" (UID: "e2736c90-d7e3-4aa1-81dd-45dea43e3b3f"). InnerVolumeSpecName "kube-api-access-wbrv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:32:36.754661 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:36.754612    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbrv8\" (UniqueName: \"kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-kube-api-access-wbrv8\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:32:36.754661 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:36.754649    5053 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-provider-credentials\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:32:36.754661 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:36.754663    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:32:37.347176 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:37.347139    5053 generic.go:356] "Generic (PLEG): container finished" podID="e2736c90-d7e3-4aa1-81dd-45dea43e3b3f" containerID="6bb05c759cb939604368e270cd6f62af0644e7e4d86e5eb613617a729114acb6" exitCode=0
Nov 03 21:32:37.347374 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:37.347202    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5"
Nov 03 21:32:37.347374 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:37.347257    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5" event={"ID":"e2736c90-d7e3-4aa1-81dd-45dea43e3b3f","Type":"ContainerDied","Data":"6bb05c759cb939604368e270cd6f62af0644e7e4d86e5eb613617a729114acb6"}
Nov 03 21:32:37.347374 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:37.347327    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5" event={"ID":"e2736c90-d7e3-4aa1-81dd-45dea43e3b3f","Type":"ContainerDied","Data":"894cbbb6a883ba7ce67b109d03da8603d8e76b17f083e77aabb468d6178cb138"}
Nov 03 21:32:37.347374 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:37.347360    5053 scope.go:122] "RemoveContainer" containerID="6bb05c759cb939604368e270cd6f62af0644e7e4d86e5eb613617a729114acb6"
Nov 03 21:32:37.363212 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:37.363190    5053 scope.go:122] "RemoveContainer" containerID="6bb05c759cb939604368e270cd6f62af0644e7e4d86e5eb613617a729114acb6"
Nov 03 21:32:37.363627 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:32:37.363601    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bb05c759cb939604368e270cd6f62af0644e7e4d86e5eb613617a729114acb6\": container with ID starting with 6bb05c759cb939604368e270cd6f62af0644e7e4d86e5eb613617a729114acb6 not found: ID does not exist" containerID="6bb05c759cb939604368e270cd6f62af0644e7e4d86e5eb613617a729114acb6"
Nov 03 21:32:37.363684 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:37.363640    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb05c759cb939604368e270cd6f62af0644e7e4d86e5eb613617a729114acb6"} err="failed to get container status \"6bb05c759cb939604368e270cd6f62af0644e7e4d86e5eb613617a729114acb6\": rpc error: code = NotFound desc = could not find container \"6bb05c759cb939604368e270cd6f62af0644e7e4d86e5eb613617a729114acb6\": container with ID starting with 6bb05c759cb939604368e270cd6f62af0644e7e4d86e5eb613617a729114acb6 not found: ID does not exist"
Nov 03 21:32:37.512378 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:37.512345    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5"]
Nov 03 21:32:37.561459 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:37.561409    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-cb5b88ddd-qqqh5"]
Nov 03 21:32:37.669087 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:37.669002    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2736c90-d7e3-4aa1-81dd-45dea43e3b3f" path="/var/lib/kubelet/pods/e2736c90-d7e3-4aa1-81dd-45dea43e3b3f/volumes"
Nov 03 21:32:40.933256 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:32:40.932575    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:32:42.958020 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:42.957982    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"]
Nov 03 21:32:42.958593 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:42.958578    5053 cpu_manager.go:414] "containerMap: removing container" podUID="bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296" containerName="payload-processing"
Nov 03 21:32:42.958646 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:42.958596    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296" containerName="payload-processing"
Nov 03 21:32:42.958646 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:42.958621    5053 cpu_manager.go:414] "containerMap: removing container" podUID="e2736c90-d7e3-4aa1-81dd-45dea43e3b3f" containerName="payload-processing"
Nov 03 21:32:42.958646 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:42.958626    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2736c90-d7e3-4aa1-81dd-45dea43e3b3f" containerName="payload-processing"
Nov 03 21:32:42.958743 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:42.958719    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbc0f4d-eb3b-4057-9ae6-9b6ce6ce3296" containerName="payload-processing"
Nov 03 21:32:42.958743 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:42.958730    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2736c90-d7e3-4aa1-81dd-45dea43e3b3f" containerName="payload-processing"
Nov 03 21:32:42.963409 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:42.963390    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.068893 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.068856    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"]
Nov 03 21:32:43.078542 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.078502    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.078729 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.078601    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.078729 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.078709    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbvl8\" (UniqueName: \"kubernetes.io/projected/74369a1d-5e9b-47a7-8a35-a6d2777e620b-kube-api-access-bbvl8\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.078855 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.078771    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.078855 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.078799    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.078971 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.078873    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.078971 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.078932    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.079072 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.078997    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.079072 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.079033    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.180735 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.180682    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.180735 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.180743    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.180956 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.180822    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.180956 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.180873    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.181041 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.180952    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbvl8\" (UniqueName: \"kubernetes.io/projected/74369a1d-5e9b-47a7-8a35-a6d2777e620b-kube-api-access-bbvl8\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.181041 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.181007    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.181041 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.181031    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.181182 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.181093    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.181182 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.181141    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.181338 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.181306    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.181459 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.181432    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.181572 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.181550    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.181637 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.181592    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.181637 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.181623    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.183573 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.183544    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.183681 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.183625    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.237775 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.237708    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.237921 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.237857    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbvl8\" (UniqueName: \"kubernetes.io/projected/74369a1d-5e9b-47a7-8a35-a6d2777e620b-kube-api-access-bbvl8\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.276469 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.276436    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:43.704228 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.704199    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"]
Nov 03 21:32:43.704972 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:32:43.704948    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74369a1d_5e9b_47a7_8a35_a6d2777e620b.slice/crio-5b4aa4ed5c70f1b20351478543dba05e3261a39af6fb6bf6b40064c0a42da064 WatchSource:0}: Error finding container 5b4aa4ed5c70f1b20351478543dba05e3261a39af6fb6bf6b40064c0a42da064: Status 404 returned error can't find the container with id 5b4aa4ed5c70f1b20351478543dba05e3261a39af6fb6bf6b40064c0a42da064
Nov 03 21:32:43.707915 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.707882    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:32:43.707977 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.707951    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:32:43.708025 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:43.707980    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:32:44.454906 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:44.454876    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx" event={"ID":"74369a1d-5e9b-47a7-8a35-a6d2777e620b","Type":"ContainerStarted","Data":"9d25c2cdc544d9e07eb42edca7dcfdbf5c921ab29ce911d63a1cb3c6d6afd6e7"}
Nov 03 21:32:44.455344 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:44.454925    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx" event={"ID":"74369a1d-5e9b-47a7-8a35-a6d2777e620b","Type":"ContainerStarted","Data":"5b4aa4ed5c70f1b20351478543dba05e3261a39af6fb6bf6b40064c0a42da064"}
Nov 03 21:32:44.670348 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:44.670301    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx" podStartSLOduration=2.670283216 podStartE2EDuration="2.670283216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:32:44.620201003 +0000 UTC m=+1079.114122916" watchObservedRunningTime="2026-11-03 21:32:44.670283216 +0000 UTC m=+1079.164205091"
Nov 03 21:32:45.276882 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:45.276850    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:45.278786 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:45.278757    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 21:32:45.278941 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:45.278818    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx" podUID="74369a1d-5e9b-47a7-8a35-a6d2777e620b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 21:32:45.929559 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:45.929529    5053 scope.go:122] "RemoveContainer" containerID="29ca2d4c063643916c488d76af3018d92e4feeee01d432e6237fa54d6c9e1140"
Nov 03 21:32:45.940909 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:45.940884    5053 scope.go:122] "RemoveContainer" containerID="129ca72db469e4742b67f9f351847a61c21e7ab4dd3483a6060b0cbded223864"
Nov 03 21:32:45.951025 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:45.951006    5053 scope.go:122] "RemoveContainer" containerID="c76669506c3d0997a536de5ebba7c56595ac80351ffd3fd517fa3ad9ff920bb6"
Nov 03 21:32:45.959627 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:45.959609    5053 scope.go:122] "RemoveContainer" containerID="0786710ff2ec8795d4510303da124d048d8fd58613256ff03ab0ff6d43590eb7"
Nov 03 21:32:45.968418 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:45.968397    5053 scope.go:122] "RemoveContainer" containerID="631e8bd80dedcb0543220a108644ee40a07bce2dfe0ed35035afc41c38c27f0a"
Nov 03 21:32:46.276843 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:46.276762    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 21:32:46.276843 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:46.276811    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx" podUID="74369a1d-5e9b-47a7-8a35-a6d2777e620b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 21:32:47.277434 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:47.277391    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 21:32:47.277923 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:47.277462    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx" podUID="74369a1d-5e9b-47a7-8a35-a6d2777e620b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 21:32:47.838152 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:47.838113    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"]
Nov 03 21:32:47.845123 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:47.845095    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:32:47.858797 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:47.858764    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4803807f-9312-40ac-badb-00c00d3f3bd3-tls-certs\") pod \"distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6\" (UID: \"4803807f-9312-40ac-badb-00c00d3f3bd3\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:32:47.858968 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:47.858912    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvn7g\" (UniqueName: \"kubernetes.io/projected/4803807f-9312-40ac-badb-00c00d3f3bd3-kube-api-access-wvn7g\") pod \"distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6\" (UID: \"4803807f-9312-40ac-badb-00c00d3f3bd3\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:32:47.858968 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:47.858945    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-model-cache\") pod \"distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6\" (UID: \"4803807f-9312-40ac-badb-00c00d3f3bd3\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:32:47.859097 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:47.858988    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-home\") pod \"distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6\" (UID: \"4803807f-9312-40ac-badb-00c00d3f3bd3\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:32:47.859097 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:47.859065    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-dshm\") pod \"distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6\" (UID: \"4803807f-9312-40ac-badb-00c00d3f3bd3\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:32:47.859214 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:47.859108    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-tmp-dir\") pod \"distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6\" (UID: \"4803807f-9312-40ac-badb-00c00d3f3bd3\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:32:47.892875 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:47.892836    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"]
Nov 03 21:32:47.938391 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:47.938349    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6de8e6\"/\"distinct-2-w4-6de8e6-kserve-self-signed-certs\""
Nov 03 21:32:47.959945 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:47.959908    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvn7g\" (UniqueName: \"kubernetes.io/projected/4803807f-9312-40ac-badb-00c00d3f3bd3-kube-api-access-wvn7g\") pod \"distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6\" (UID: \"4803807f-9312-40ac-badb-00c00d3f3bd3\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:32:47.959945 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:47.959948    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-model-cache\") pod \"distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6\" (UID: \"4803807f-9312-40ac-badb-00c00d3f3bd3\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:32:47.960199 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:47.959982    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-home\") pod \"distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6\" (UID: \"4803807f-9312-40ac-badb-00c00d3f3bd3\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:32:47.960199 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:47.960005    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-dshm\") pod \"distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6\" (UID: \"4803807f-9312-40ac-badb-00c00d3f3bd3\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:32:47.960199 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:47.960026    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-tmp-dir\") pod \"distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6\" (UID: \"4803807f-9312-40ac-badb-00c00d3f3bd3\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:32:47.960199 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:47.960054    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4803807f-9312-40ac-badb-00c00d3f3bd3-tls-certs\") pod \"distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6\" (UID: \"4803807f-9312-40ac-badb-00c00d3f3bd3\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:32:47.960467 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:47.960443    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-home\") pod \"distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6\" (UID: \"4803807f-9312-40ac-badb-00c00d3f3bd3\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:32:47.960654 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:47.960539    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-model-cache\") pod \"distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6\" (UID: \"4803807f-9312-40ac-badb-00c00d3f3bd3\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:32:47.960932 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:47.960549    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-tmp-dir\") pod \"distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6\" (UID: \"4803807f-9312-40ac-badb-00c00d3f3bd3\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:32:47.962578 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:47.962558    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-dshm\") pod \"distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6\" (UID: \"4803807f-9312-40ac-badb-00c00d3f3bd3\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:32:47.962847 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:47.962830    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4803807f-9312-40ac-badb-00c00d3f3bd3-tls-certs\") pod \"distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6\" (UID: \"4803807f-9312-40ac-badb-00c00d3f3bd3\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:32:48.047899 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:48.047869    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvn7g\" (UniqueName: \"kubernetes.io/projected/4803807f-9312-40ac-badb-00c00d3f3bd3-kube-api-access-wvn7g\") pod \"distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6\" (UID: \"4803807f-9312-40ac-badb-00c00d3f3bd3\") " pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:32:48.163900 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:48.163806    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:32:48.279588 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:48.277302    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 21:32:48.279588 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:48.277392    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx" podUID="74369a1d-5e9b-47a7-8a35-a6d2777e620b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 21:32:48.620261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:48.617382    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"]
Nov 03 21:32:49.277349 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:49.277313    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 21:32:49.277533 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:49.277370    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx" podUID="74369a1d-5e9b-47a7-8a35-a6d2777e620b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 21:32:49.491216 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:49.491178    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6" event={"ID":"4803807f-9312-40ac-badb-00c00d3f3bd3","Type":"ContainerStarted","Data":"57791de81b25257bb1d5c63321a87dce0bf674c5fc02a0ea5fe43ed6f0a5c8a5"}
Nov 03 21:32:49.491619 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:49.491228    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6" event={"ID":"4803807f-9312-40ac-badb-00c00d3f3bd3","Type":"ContainerStarted","Data":"1352a7cde22407f8365e63dfa12cd6a98886dca53d57411a1540132f31c4e13a"}
Nov 03 21:32:49.627073 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:49.626954    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6" podStartSLOduration=2.626935949 podStartE2EDuration="2.626935949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:32:49.604605761 +0000 UTC m=+1084.098527638" watchObservedRunningTime="2026-11-03 21:32:49.626935949 +0000 UTC m=+1084.120857827"
Nov 03 21:32:50.164449 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:50.164413    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:32:50.277704 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:50.277670    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 21:32:50.277909 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:50.277728    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx" podUID="74369a1d-5e9b-47a7-8a35-a6d2777e620b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 21:32:51.281979 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:51.281950    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:51.282453 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:51.282438    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:51.283159 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:51.283141    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:32:58.164084 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:58.164047    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:32:58.176616 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:58.176590    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:32:58.575336 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:32:58.575303    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:33:07.371291 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:07.371230    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd75a1ed-59df5"]
Nov 03 21:33:07.375444 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:07.375420    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd75a1ed-59df5"
Nov 03 21:33:07.431284 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:07.431253    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd75a1ed-59df5"]
Nov 03 21:33:07.477293 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:07.477202    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:33:07.477293 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:07.477264    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:33:07.477293 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:07.477203    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:33:07.477584 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:07.477443    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:33:07.503202 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:07.503163    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcxws\" (UniqueName: \"kubernetes.io/projected/f572ca4a-00b0-43b2-9681-91680b27a69b-kube-api-access-wcxws\") pod \"maas-api-revoke-sub-fd75a1ed-59df5\" (UID: \"f572ca4a-00b0-43b2-9681-91680b27a69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd75a1ed-59df5"
Nov 03 21:33:07.503398 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:07.503218    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f572ca4a-00b0-43b2-9681-91680b27a69b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd75a1ed-59df5\" (UID: \"f572ca4a-00b0-43b2-9681-91680b27a69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd75a1ed-59df5"
Nov 03 21:33:07.503454 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:07.503410    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f572ca4a-00b0-43b2-9681-91680b27a69b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd75a1ed-59df5\" (UID: \"f572ca4a-00b0-43b2-9681-91680b27a69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd75a1ed-59df5"
Nov 03 21:33:07.604419 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:07.604388    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f572ca4a-00b0-43b2-9681-91680b27a69b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd75a1ed-59df5\" (UID: \"f572ca4a-00b0-43b2-9681-91680b27a69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd75a1ed-59df5"
Nov 03 21:33:07.604623 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:07.604525    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f572ca4a-00b0-43b2-9681-91680b27a69b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd75a1ed-59df5\" (UID: \"f572ca4a-00b0-43b2-9681-91680b27a69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd75a1ed-59df5"
Nov 03 21:33:07.604623 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:07.604614    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcxws\" (UniqueName: \"kubernetes.io/projected/f572ca4a-00b0-43b2-9681-91680b27a69b-kube-api-access-wcxws\") pod \"maas-api-revoke-sub-fd75a1ed-59df5\" (UID: \"f572ca4a-00b0-43b2-9681-91680b27a69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd75a1ed-59df5"
Nov 03 21:33:07.605346 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:07.605321    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f572ca4a-00b0-43b2-9681-91680b27a69b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd75a1ed-59df5\" (UID: \"f572ca4a-00b0-43b2-9681-91680b27a69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd75a1ed-59df5"
Nov 03 21:33:07.606992 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:07.606971    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f572ca4a-00b0-43b2-9681-91680b27a69b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd75a1ed-59df5\" (UID: \"f572ca4a-00b0-43b2-9681-91680b27a69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd75a1ed-59df5"
Nov 03 21:33:07.662026 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:07.661956    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcxws\" (UniqueName: \"kubernetes.io/projected/f572ca4a-00b0-43b2-9681-91680b27a69b-kube-api-access-wcxws\") pod \"maas-api-revoke-sub-fd75a1ed-59df5\" (UID: \"f572ca4a-00b0-43b2-9681-91680b27a69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd75a1ed-59df5"
Nov 03 21:33:07.691371 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:07.691339    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd75a1ed-59df5"
Nov 03 21:33:08.072438 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:08.072409    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd75a1ed-59df5"]
Nov 03 21:33:08.073325 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:33:08.073301    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf572ca4a_00b0_43b2_9681_91680b27a69b.slice/crio-d1627cbc083865dcf2ad514a8f1533c9dba2800d54e1546aee888abd199e3106 WatchSource:0}: Error finding container d1627cbc083865dcf2ad514a8f1533c9dba2800d54e1546aee888abd199e3106: Status 404 returned error can't find the container with id d1627cbc083865dcf2ad514a8f1533c9dba2800d54e1546aee888abd199e3106
Nov 03 21:33:08.650177 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:08.650087    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd75a1ed-59df5" event={"ID":"f572ca4a-00b0-43b2-9681-91680b27a69b","Type":"ContainerStarted","Data":"56fb26aad760ca958f111bb5fe00a34b60dd0f022c1dd161a32705578c599e75"}
Nov 03 21:33:08.650177 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:08.650143    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd75a1ed-59df5" event={"ID":"f572ca4a-00b0-43b2-9681-91680b27a69b","Type":"ContainerStarted","Data":"d1627cbc083865dcf2ad514a8f1533c9dba2800d54e1546aee888abd199e3106"}
Nov 03 21:33:08.764178 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:08.764122    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd75a1ed-59df5" podStartSLOduration=1.7641058219999999 podStartE2EDuration="1.764105822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:33:08.763051123 +0000 UTC m=+1103.256973000" watchObservedRunningTime="2026-11-03 21:33:08.764105822 +0000 UTC m=+1103.258027697"
Nov 03 21:33:09.659559 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:09.659522    5053 generic.go:356] "Generic (PLEG): container finished" podID="f572ca4a-00b0-43b2-9681-91680b27a69b" containerID="56fb26aad760ca958f111bb5fe00a34b60dd0f022c1dd161a32705578c599e75" exitCode=0
Nov 03 21:33:09.676601 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:09.676566    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd75a1ed-59df5" event={"ID":"f572ca4a-00b0-43b2-9681-91680b27a69b","Type":"ContainerDied","Data":"56fb26aad760ca958f111bb5fe00a34b60dd0f022c1dd161a32705578c599e75"}
Nov 03 21:33:10.834114 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:10.834084    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd75a1ed-59df5"
Nov 03 21:33:10.933349 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:33:10.933229    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:33:10.945766 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:10.945731    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f572ca4a-00b0-43b2-9681-91680b27a69b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f572ca4a-00b0-43b2-9681-91680b27a69b-maas-api-xks-ca\") pod \"f572ca4a-00b0-43b2-9681-91680b27a69b\" (UID: \"f572ca4a-00b0-43b2-9681-91680b27a69b\") "
Nov 03 21:33:10.945957 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:10.945859    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f572ca4a-00b0-43b2-9681-91680b27a69b-kube-api-access-wcxws\" (UniqueName: \"kubernetes.io/projected/f572ca4a-00b0-43b2-9681-91680b27a69b-kube-api-access-wcxws\") pod \"f572ca4a-00b0-43b2-9681-91680b27a69b\" (UID: \"f572ca4a-00b0-43b2-9681-91680b27a69b\") "
Nov 03 21:33:10.945957 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:10.945891    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f572ca4a-00b0-43b2-9681-91680b27a69b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f572ca4a-00b0-43b2-9681-91680b27a69b-maas-api-openshift-ca\") pod \"f572ca4a-00b0-43b2-9681-91680b27a69b\" (UID: \"f572ca4a-00b0-43b2-9681-91680b27a69b\") "
Nov 03 21:33:10.946901 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:10.946877    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f572ca4a-00b0-43b2-9681-91680b27a69b-maas-api-openshift-ca" pod "f572ca4a-00b0-43b2-9681-91680b27a69b" (UID: "f572ca4a-00b0-43b2-9681-91680b27a69b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:33:10.948934 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:10.948820    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f572ca4a-00b0-43b2-9681-91680b27a69b-kube-api-access-wcxws" pod "f572ca4a-00b0-43b2-9681-91680b27a69b" (UID: "f572ca4a-00b0-43b2-9681-91680b27a69b"). InnerVolumeSpecName "kube-api-access-wcxws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:33:10.955304 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:10.955273    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f572ca4a-00b0-43b2-9681-91680b27a69b-maas-api-xks-ca" pod "f572ca4a-00b0-43b2-9681-91680b27a69b" (UID: "f572ca4a-00b0-43b2-9681-91680b27a69b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:33:11.047813 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.047776    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcxws\" (UniqueName: \"kubernetes.io/projected/f572ca4a-00b0-43b2-9681-91680b27a69b-kube-api-access-wcxws\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:11.047813 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.047809    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f572ca4a-00b0-43b2-9681-91680b27a69b-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:11.047813 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.047821    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f572ca4a-00b0-43b2-9681-91680b27a69b-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:11.523339 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.523302    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"]
Nov 03 21:33:11.524199 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.524179    5053 cpu_manager.go:414] "containerMap: removing container" podUID="f572ca4a-00b0-43b2-9681-91680b27a69b" containerName="revoke-keys"
Nov 03 21:33:11.524284 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.524202    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f572ca4a-00b0-43b2-9681-91680b27a69b" containerName="revoke-keys"
Nov 03 21:33:11.524398 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.524386    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="f572ca4a-00b0-43b2-9681-91680b27a69b" containerName="revoke-keys"
Nov 03 21:33:11.528567 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.528544    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:11.582417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.582388    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"]
Nov 03 21:33:11.627473 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.627439    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-da5f0b\"/\"kube-root-ca.crt\""
Nov 03 21:33:11.627674 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.627541    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-da5f0b\"/\"openshift-service-ca.crt\""
Nov 03 21:33:11.627674 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.627633    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-da5f0b\"/\"default-dockercfg-cmzvh\""
Nov 03 21:33:11.630508 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.630472    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-da5f0b\"/\"distinct-w3-da5f0b-kserve-self-signed-certs\""
Nov 03 21:33:11.655814 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.655782    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-tls-certs\") pod \"distinct-w3-da5f0b-kserve-8647556fc6-zsrmd\" (UID: \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:11.655814 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.655816    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-dshm\") pod \"distinct-w3-da5f0b-kserve-8647556fc6-zsrmd\" (UID: \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:11.656017 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.655983    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-tmp-dir\") pod \"distinct-w3-da5f0b-kserve-8647556fc6-zsrmd\" (UID: \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:11.656069 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.656035    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-home\") pod \"distinct-w3-da5f0b-kserve-8647556fc6-zsrmd\" (UID: \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:11.656137 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.656125    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxk69\" (UniqueName: \"kubernetes.io/projected/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-kube-api-access-vxk69\") pod \"distinct-w3-da5f0b-kserve-8647556fc6-zsrmd\" (UID: \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:11.656172 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.656152    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-model-cache\") pod \"distinct-w3-da5f0b-kserve-8647556fc6-zsrmd\" (UID: \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:11.674672 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.674646    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd75a1ed-59df5"
Nov 03 21:33:11.674839 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.674684    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd75a1ed-59df5" event={"ID":"f572ca4a-00b0-43b2-9681-91680b27a69b","Type":"ContainerDied","Data":"d1627cbc083865dcf2ad514a8f1533c9dba2800d54e1546aee888abd199e3106"}
Nov 03 21:33:11.674839 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.674742    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1627cbc083865dcf2ad514a8f1533c9dba2800d54e1546aee888abd199e3106"
Nov 03 21:33:11.757008 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.756970    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxk69\" (UniqueName: \"kubernetes.io/projected/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-kube-api-access-vxk69\") pod \"distinct-w3-da5f0b-kserve-8647556fc6-zsrmd\" (UID: \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:11.757208 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.757016    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-model-cache\") pod \"distinct-w3-da5f0b-kserve-8647556fc6-zsrmd\" (UID: \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:11.757208 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.757100    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-tls-certs\") pod \"distinct-w3-da5f0b-kserve-8647556fc6-zsrmd\" (UID: \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:11.757208 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.757121    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-dshm\") pod \"distinct-w3-da5f0b-kserve-8647556fc6-zsrmd\" (UID: \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:11.757400 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.757229    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-tmp-dir\") pod \"distinct-w3-da5f0b-kserve-8647556fc6-zsrmd\" (UID: \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:11.757400 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.757339    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-home\") pod \"distinct-w3-da5f0b-kserve-8647556fc6-zsrmd\" (UID: \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:11.757622 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.757596    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-model-cache\") pod \"distinct-w3-da5f0b-kserve-8647556fc6-zsrmd\" (UID: \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:11.757722 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.757699    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-tmp-dir\") pod \"distinct-w3-da5f0b-kserve-8647556fc6-zsrmd\" (UID: \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:11.757782 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.757749    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-home\") pod \"distinct-w3-da5f0b-kserve-8647556fc6-zsrmd\" (UID: \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:11.759665 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.759644    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-dshm\") pod \"distinct-w3-da5f0b-kserve-8647556fc6-zsrmd\" (UID: \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:11.759887 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.759871    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-tls-certs\") pod \"distinct-w3-da5f0b-kserve-8647556fc6-zsrmd\" (UID: \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:11.813072 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.812996    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxk69\" (UniqueName: \"kubernetes.io/projected/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-kube-api-access-vxk69\") pod \"distinct-w3-da5f0b-kserve-8647556fc6-zsrmd\" (UID: \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:11.843468 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:11.843429    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:12.231871 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:12.231829    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"]
Nov 03 21:33:12.232596 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:33:12.232559    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82077ea7_1e05_4b6e_806b_a2d2d2a64c4f.slice/crio-24a7be786893ed61311da1041dd84996fc16c8f5e805b3dcd580d604da3a979d WatchSource:0}: Error finding container 24a7be786893ed61311da1041dd84996fc16c8f5e805b3dcd580d604da3a979d: Status 404 returned error can't find the container with id 24a7be786893ed61311da1041dd84996fc16c8f5e805b3dcd580d604da3a979d
Nov 03 21:33:12.683699 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:12.683612    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd" event={"ID":"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f","Type":"ContainerStarted","Data":"55a662149df6a05773b532e8fde81b5a9c9e00ea08f18a093f3581e57e2f513f"}
Nov 03 21:33:12.683699 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:12.683662    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd" event={"ID":"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f","Type":"ContainerStarted","Data":"24a7be786893ed61311da1041dd84996fc16c8f5e805b3dcd580d604da3a979d"}
Nov 03 21:33:12.801614 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:12.801563    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd" podStartSLOduration=1.801546356 podStartE2EDuration="1.801546356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:33:12.797823594 +0000 UTC m=+1107.291745471" watchObservedRunningTime="2026-11-03 21:33:12.801546356 +0000 UTC m=+1107.295468230"
Nov 03 21:33:12.844216 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:12.844177    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:21.844116 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:21.844084    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:21.856450 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:21.856426    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:21.908207 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:21.908178    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8"]
Nov 03 21:33:21.913639 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:21.913612    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8"
Nov 03 21:33:21.985440 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:21.985411    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8"]
Nov 03 21:33:22.015744 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.015709    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bffc29d5-d3bb-429c-8772-7b672dd1f25f-routing-overlay-volume\") pod \"payload-processing-external-model-6575949dfc-ntpn8\" (UID: \"bffc29d5-d3bb-429c-8772-7b672dd1f25f\") " pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8"
Nov 03 21:33:22.015932 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.015789    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57pmx\" (UniqueName: \"kubernetes.io/projected/bffc29d5-d3bb-429c-8772-7b672dd1f25f-kube-api-access-57pmx\") pod \"payload-processing-external-model-6575949dfc-ntpn8\" (UID: \"bffc29d5-d3bb-429c-8772-7b672dd1f25f\") " pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8"
Nov 03 21:33:22.015932 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.015833    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bffc29d5-d3bb-429c-8772-7b672dd1f25f-plugins-config-volume\") pod \"payload-processing-external-model-6575949dfc-ntpn8\" (UID: \"bffc29d5-d3bb-429c-8772-7b672dd1f25f\") " pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8"
Nov 03 21:33:22.016106 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.016084    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bffc29d5-d3bb-429c-8772-7b672dd1f25f-provider-credentials\") pod \"payload-processing-external-model-6575949dfc-ntpn8\" (UID: \"bffc29d5-d3bb-429c-8772-7b672dd1f25f\") " pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8"
Nov 03 21:33:22.117170 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.117081    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bffc29d5-d3bb-429c-8772-7b672dd1f25f-routing-overlay-volume\") pod \"payload-processing-external-model-6575949dfc-ntpn8\" (UID: \"bffc29d5-d3bb-429c-8772-7b672dd1f25f\") " pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8"
Nov 03 21:33:22.117170 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.117149    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57pmx\" (UniqueName: \"kubernetes.io/projected/bffc29d5-d3bb-429c-8772-7b672dd1f25f-kube-api-access-57pmx\") pod \"payload-processing-external-model-6575949dfc-ntpn8\" (UID: \"bffc29d5-d3bb-429c-8772-7b672dd1f25f\") " pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8"
Nov 03 21:33:22.117419 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.117177    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bffc29d5-d3bb-429c-8772-7b672dd1f25f-plugins-config-volume\") pod \"payload-processing-external-model-6575949dfc-ntpn8\" (UID: \"bffc29d5-d3bb-429c-8772-7b672dd1f25f\") " pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8"
Nov 03 21:33:22.117419 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.117404    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bffc29d5-d3bb-429c-8772-7b672dd1f25f-provider-credentials\") pod \"payload-processing-external-model-6575949dfc-ntpn8\" (UID: \"bffc29d5-d3bb-429c-8772-7b672dd1f25f\") " pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8"
Nov 03 21:33:22.117852 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.117826    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bffc29d5-d3bb-429c-8772-7b672dd1f25f-routing-overlay-volume\") pod \"payload-processing-external-model-6575949dfc-ntpn8\" (UID: \"bffc29d5-d3bb-429c-8772-7b672dd1f25f\") " pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8"
Nov 03 21:33:22.117988 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.117924    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bffc29d5-d3bb-429c-8772-7b672dd1f25f-plugins-config-volume\") pod \"payload-processing-external-model-6575949dfc-ntpn8\" (UID: \"bffc29d5-d3bb-429c-8772-7b672dd1f25f\") " pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8"
Nov 03 21:33:22.119945 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.119920    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bffc29d5-d3bb-429c-8772-7b672dd1f25f-provider-credentials\") pod \"payload-processing-external-model-6575949dfc-ntpn8\" (UID: \"bffc29d5-d3bb-429c-8772-7b672dd1f25f\") " pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8"
Nov 03 21:33:22.179271 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.179198    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57pmx\" (UniqueName: \"kubernetes.io/projected/bffc29d5-d3bb-429c-8772-7b672dd1f25f-kube-api-access-57pmx\") pod \"payload-processing-external-model-6575949dfc-ntpn8\" (UID: \"bffc29d5-d3bb-429c-8772-7b672dd1f25f\") " pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8"
Nov 03 21:33:22.228493 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.228460    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8"
Nov 03 21:33:22.434773 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.434683    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2d3ea98-zcqh9"]
Nov 03 21:33:22.439054 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.439028    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d3ea98-zcqh9"
Nov 03 21:33:22.491852 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.491697    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:33:22.491852 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.491695    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:33:22.491852 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.491753    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:33:22.492124 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.491906    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:33:22.495614 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.495585    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2d3ea98-zcqh9"]
Nov 03 21:33:22.521612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.521579    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7mk7\" (UniqueName: \"kubernetes.io/projected/2f120939-31ab-4bc2-8773-42fbd6268939-kube-api-access-l7mk7\") pod \"maas-api-revoke-sub-d2d3ea98-zcqh9\" (UID: \"2f120939-31ab-4bc2-8773-42fbd6268939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d3ea98-zcqh9"
Nov 03 21:33:22.521824 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.521740    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f120939-31ab-4bc2-8773-42fbd6268939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2d3ea98-zcqh9\" (UID: \"2f120939-31ab-4bc2-8773-42fbd6268939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d3ea98-zcqh9"
Nov 03 21:33:22.521824 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.521792    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f120939-31ab-4bc2-8773-42fbd6268939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2d3ea98-zcqh9\" (UID: \"2f120939-31ab-4bc2-8773-42fbd6268939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d3ea98-zcqh9"
Nov 03 21:33:22.605260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.605219    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8"]
Nov 03 21:33:22.606426 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:33:22.606394    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbffc29d5_d3bb_429c_8772_7b672dd1f25f.slice/crio-62c879371234f4cacabca20fffe203dbeb8a4a1289dcc71b4aaad0c2c0eb1959 WatchSource:0}: Error finding container 62c879371234f4cacabca20fffe203dbeb8a4a1289dcc71b4aaad0c2c0eb1959: Status 404 returned error can't find the container with id 62c879371234f4cacabca20fffe203dbeb8a4a1289dcc71b4aaad0c2c0eb1959
Nov 03 21:33:22.623472 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.623430    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7mk7\" (UniqueName: \"kubernetes.io/projected/2f120939-31ab-4bc2-8773-42fbd6268939-kube-api-access-l7mk7\") pod \"maas-api-revoke-sub-d2d3ea98-zcqh9\" (UID: \"2f120939-31ab-4bc2-8773-42fbd6268939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d3ea98-zcqh9"
Nov 03 21:33:22.623539 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.623528    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f120939-31ab-4bc2-8773-42fbd6268939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2d3ea98-zcqh9\" (UID: \"2f120939-31ab-4bc2-8773-42fbd6268939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d3ea98-zcqh9"
Nov 03 21:33:22.623577 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.623553    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f120939-31ab-4bc2-8773-42fbd6268939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2d3ea98-zcqh9\" (UID: \"2f120939-31ab-4bc2-8773-42fbd6268939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d3ea98-zcqh9"
Nov 03 21:33:22.624134 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.624109    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f120939-31ab-4bc2-8773-42fbd6268939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2d3ea98-zcqh9\" (UID: \"2f120939-31ab-4bc2-8773-42fbd6268939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d3ea98-zcqh9"
Nov 03 21:33:22.625992 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.625969    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f120939-31ab-4bc2-8773-42fbd6268939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2d3ea98-zcqh9\" (UID: \"2f120939-31ab-4bc2-8773-42fbd6268939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d3ea98-zcqh9"
Nov 03 21:33:22.679565 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.679525    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7mk7\" (UniqueName: \"kubernetes.io/projected/2f120939-31ab-4bc2-8773-42fbd6268939-kube-api-access-l7mk7\") pod \"maas-api-revoke-sub-d2d3ea98-zcqh9\" (UID: \"2f120939-31ab-4bc2-8773-42fbd6268939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d3ea98-zcqh9"
Nov 03 21:33:22.748108 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.748070    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8" event={"ID":"bffc29d5-d3bb-429c-8772-7b672dd1f25f","Type":"ContainerStarted","Data":"c20fa0b8e1e0cde2c4e0695acf38f936367dc1eae836dafea79d23b2e089016a"}
Nov 03 21:33:22.748369 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.748137    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8" event={"ID":"bffc29d5-d3bb-429c-8772-7b672dd1f25f","Type":"ContainerStarted","Data":"62c879371234f4cacabca20fffe203dbeb8a4a1289dcc71b4aaad0c2c0eb1959"}
Nov 03 21:33:22.748647 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.748438    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8"
Nov 03 21:33:22.753778 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.753756    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d3ea98-zcqh9"
Nov 03 21:33:22.759577 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.759561    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:22.861267 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:22.861186    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8" podStartSLOduration=1.86117082 podStartE2EDuration="1.86117082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:33:22.860180007 +0000 UTC m=+1117.354101882" watchObservedRunningTime="2026-11-03 21:33:22.86117082 +0000 UTC m=+1117.355092694"
Nov 03 21:33:23.134154 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:23.134119    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2d3ea98-zcqh9"]
Nov 03 21:33:23.134374 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:33:23.134271    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f120939_31ab_4bc2_8773_42fbd6268939.slice/crio-0c19e1a2bb0c02d486d84995ddff3ab4b62b62052c7c7295903f47b1b4b15503 WatchSource:0}: Error finding container 0c19e1a2bb0c02d486d84995ddff3ab4b62b62052c7c7295903f47b1b4b15503: Status 404 returned error can't find the container with id 0c19e1a2bb0c02d486d84995ddff3ab4b62b62052c7c7295903f47b1b4b15503
Nov 03 21:33:23.757368 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:23.757328    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d3ea98-zcqh9" event={"ID":"2f120939-31ab-4bc2-8773-42fbd6268939","Type":"ContainerStarted","Data":"89637fd5592a5e0eaa04e9fbb9bb530577b22a6388c824186662b4f5ac8c77eb"}
Nov 03 21:33:23.757583 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:23.757399    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d3ea98-zcqh9" event={"ID":"2f120939-31ab-4bc2-8773-42fbd6268939","Type":"ContainerStarted","Data":"0c19e1a2bb0c02d486d84995ddff3ab4b62b62052c7c7295903f47b1b4b15503"}
Nov 03 21:33:23.870956 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:23.870906    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d3ea98-zcqh9" podStartSLOduration=1.8708915099999999 podStartE2EDuration="1.87089151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:33:23.868934015 +0000 UTC m=+1118.362855892" watchObservedRunningTime="2026-11-03 21:33:23.87089151 +0000 UTC m=+1118.364813385"
Nov 03 21:33:24.766149 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:24.766118    5053 generic.go:356] "Generic (PLEG): container finished" podID="2f120939-31ab-4bc2-8773-42fbd6268939" containerID="89637fd5592a5e0eaa04e9fbb9bb530577b22a6388c824186662b4f5ac8c77eb" exitCode=0
Nov 03 21:33:24.766349 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:24.766176    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d3ea98-zcqh9" event={"ID":"2f120939-31ab-4bc2-8773-42fbd6268939","Type":"ContainerDied","Data":"89637fd5592a5e0eaa04e9fbb9bb530577b22a6388c824186662b4f5ac8c77eb"}
Nov 03 21:33:25.949573 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:25.949544    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d3ea98-zcqh9"
Nov 03 21:33:26.068582 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:26.068493    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f120939-31ab-4bc2-8773-42fbd6268939-kube-api-access-l7mk7\" (UniqueName: \"kubernetes.io/projected/2f120939-31ab-4bc2-8773-42fbd6268939-kube-api-access-l7mk7\") pod \"2f120939-31ab-4bc2-8773-42fbd6268939\" (UID: \"2f120939-31ab-4bc2-8773-42fbd6268939\") "
Nov 03 21:33:26.068762 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:26.068616    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f120939-31ab-4bc2-8773-42fbd6268939-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f120939-31ab-4bc2-8773-42fbd6268939-maas-api-openshift-ca\") pod \"2f120939-31ab-4bc2-8773-42fbd6268939\" (UID: \"2f120939-31ab-4bc2-8773-42fbd6268939\") "
Nov 03 21:33:26.068762 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:26.068676    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f120939-31ab-4bc2-8773-42fbd6268939-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f120939-31ab-4bc2-8773-42fbd6268939-maas-api-xks-ca\") pod \"2f120939-31ab-4bc2-8773-42fbd6268939\" (UID: \"2f120939-31ab-4bc2-8773-42fbd6268939\") "
Nov 03 21:33:26.068965 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:26.068942    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f120939-31ab-4bc2-8773-42fbd6268939-maas-api-openshift-ca" pod "2f120939-31ab-4bc2-8773-42fbd6268939" (UID: "2f120939-31ab-4bc2-8773-42fbd6268939"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:33:26.070280 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:26.069165    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f120939-31ab-4bc2-8773-42fbd6268939-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:26.072270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:26.071096    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f120939-31ab-4bc2-8773-42fbd6268939-maas-api-xks-ca" pod "2f120939-31ab-4bc2-8773-42fbd6268939" (UID: "2f120939-31ab-4bc2-8773-42fbd6268939"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:33:26.072270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:26.071394    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f120939-31ab-4bc2-8773-42fbd6268939-kube-api-access-l7mk7" pod "2f120939-31ab-4bc2-8773-42fbd6268939" (UID: "2f120939-31ab-4bc2-8773-42fbd6268939"). InnerVolumeSpecName "kube-api-access-l7mk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:33:26.170181 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:26.170150    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f120939-31ab-4bc2-8773-42fbd6268939-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:26.170425 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:26.170402    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7mk7\" (UniqueName: \"kubernetes.io/projected/2f120939-31ab-4bc2-8773-42fbd6268939-kube-api-access-l7mk7\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:26.780600 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:26.780572    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d3ea98-zcqh9"
Nov 03 21:33:26.780776 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:26.780602    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d3ea98-zcqh9" event={"ID":"2f120939-31ab-4bc2-8773-42fbd6268939","Type":"ContainerDied","Data":"0c19e1a2bb0c02d486d84995ddff3ab4b62b62052c7c7295903f47b1b4b15503"}
Nov 03 21:33:26.780776 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:26.780647    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c19e1a2bb0c02d486d84995ddff3ab4b62b62052c7c7295903f47b1b4b15503"
Nov 03 21:33:33.606474 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.606436    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"]
Nov 03 21:33:33.607350 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.607330    5053 cpu_manager.go:414] "containerMap: removing container" podUID="2f120939-31ab-4bc2-8773-42fbd6268939" containerName="revoke-keys"
Nov 03 21:33:33.607350 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.607349    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f120939-31ab-4bc2-8773-42fbd6268939" containerName="revoke-keys"
Nov 03 21:33:33.607523 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.607506    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f120939-31ab-4bc2-8773-42fbd6268939" containerName="revoke-keys"
Nov 03 21:33:33.611583 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.611561    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.708507 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.708472    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.708720 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.708533    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.708720 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.708645    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.708720 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.708691    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqn26\" (UniqueName: \"kubernetes.io/projected/c43adf43-1729-4f0d-b6c7-3918c0fff604-kube-api-access-vqn26\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.708871 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.708779    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.708871 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.708839    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.708968 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.708917    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.709020 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.708973    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c43adf43-1729-4f0d-b6c7-3918c0fff604-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.709069 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.709040    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.761156 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.761115    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8" podUID="bffc29d5-d3bb-429c-8772-7b672dd1f25f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:33:33.811727 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.811689    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"]
Nov 03 21:33:33.812420 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.812391    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.812546 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.812466    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.812546 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.812512    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.813372 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.812573    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.813372 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.812620    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqn26\" (UniqueName: \"kubernetes.io/projected/c43adf43-1729-4f0d-b6c7-3918c0fff604-kube-api-access-vqn26\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.813372 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.812698    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.813372 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.812734    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.813372 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.812777    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.813372 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.812843    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c43adf43-1729-4f0d-b6c7-3918c0fff604-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.813703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.813679    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c43adf43-1729-4f0d-b6c7-3918c0fff604-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.813925 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.813905    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.818660 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.817568    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.818660 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.817811    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.818660 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.818044    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.818660 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.818598    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.821745 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.821721    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.876042 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.875964    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqn26\" (UniqueName: \"kubernetes.io/projected/c43adf43-1729-4f0d-b6c7-3918c0fff604-kube-api-access-vqn26\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.876174 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.876075    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4466cc7-rbvjl"]
Nov 03 21:33:33.876923 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.876902    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.881505 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.881488    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4466cc7-rbvjl"
Nov 03 21:33:33.914075 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.914039    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad574db0-89e0-4a72-84bf-bbd62702b536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4466cc7-rbvjl\" (UID: \"ad574db0-89e0-4a72-84bf-bbd62702b536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4466cc7-rbvjl"
Nov 03 21:33:33.914290 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.914269    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfqdz\" (UniqueName: \"kubernetes.io/projected/ad574db0-89e0-4a72-84bf-bbd62702b536-kube-api-access-dfqdz\") pod \"maas-api-revoke-sub-e4466cc7-rbvjl\" (UID: \"ad574db0-89e0-4a72-84bf-bbd62702b536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4466cc7-rbvjl"
Nov 03 21:33:33.914404 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.914389    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad574db0-89e0-4a72-84bf-bbd62702b536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4466cc7-rbvjl\" (UID: \"ad574db0-89e0-4a72-84bf-bbd62702b536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4466cc7-rbvjl"
Nov 03 21:33:33.926721 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.926697    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:33.934016 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.933989    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:33:33.934143 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.934128    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:33:33.934783 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.934761    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:33:33.934889 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.934874    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:33:33.937310 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:33.937271    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4466cc7-rbvjl"]
Nov 03 21:33:34.016818 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:34.016778    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfqdz\" (UniqueName: \"kubernetes.io/projected/ad574db0-89e0-4a72-84bf-bbd62702b536-kube-api-access-dfqdz\") pod \"maas-api-revoke-sub-e4466cc7-rbvjl\" (UID: \"ad574db0-89e0-4a72-84bf-bbd62702b536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4466cc7-rbvjl"
Nov 03 21:33:34.016968 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:34.016898    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad574db0-89e0-4a72-84bf-bbd62702b536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4466cc7-rbvjl\" (UID: \"ad574db0-89e0-4a72-84bf-bbd62702b536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4466cc7-rbvjl"
Nov 03 21:33:34.017043 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:34.017024    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad574db0-89e0-4a72-84bf-bbd62702b536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4466cc7-rbvjl\" (UID: \"ad574db0-89e0-4a72-84bf-bbd62702b536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4466cc7-rbvjl"
Nov 03 21:33:34.018057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:34.018030    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad574db0-89e0-4a72-84bf-bbd62702b536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4466cc7-rbvjl\" (UID: \"ad574db0-89e0-4a72-84bf-bbd62702b536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4466cc7-rbvjl"
Nov 03 21:33:34.024574 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:34.024541    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad574db0-89e0-4a72-84bf-bbd62702b536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4466cc7-rbvjl\" (UID: \"ad574db0-89e0-4a72-84bf-bbd62702b536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4466cc7-rbvjl"
Nov 03 21:33:34.077589 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:34.077561    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfqdz\" (UniqueName: \"kubernetes.io/projected/ad574db0-89e0-4a72-84bf-bbd62702b536-kube-api-access-dfqdz\") pod \"maas-api-revoke-sub-e4466cc7-rbvjl\" (UID: \"ad574db0-89e0-4a72-84bf-bbd62702b536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4466cc7-rbvjl"
Nov 03 21:33:34.211304 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:34.211278    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4466cc7-rbvjl"
Nov 03 21:33:34.372671 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:34.372647    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"]
Nov 03 21:33:34.372786 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:33:34.372713    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43adf43_1729_4f0d_b6c7_3918c0fff604.slice/crio-b11439c52b6c2fd009ae82c47f9319cc7bf6b61c49bb690c8d47f9a140a43f37 WatchSource:0}: Error finding container b11439c52b6c2fd009ae82c47f9319cc7bf6b61c49bb690c8d47f9a140a43f37: Status 404 returned error can't find the container with id b11439c52b6c2fd009ae82c47f9319cc7bf6b61c49bb690c8d47f9a140a43f37
Nov 03 21:33:34.375059 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:34.375031    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:33:34.375131 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:34.375098    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:33:34.375131 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:34.375125    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:33:34.588625 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:34.588598    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4466cc7-rbvjl"]
Nov 03 21:33:34.589840 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:33:34.589814    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad574db0_89e0_4a72_84bf_bbd62702b536.slice/crio-e9debf6116eee5bc811cc5d49c0e6ca03bc924ff1db688ba1773537041804442 WatchSource:0}: Error finding container e9debf6116eee5bc811cc5d49c0e6ca03bc924ff1db688ba1773537041804442: Status 404 returned error can't find the container with id e9debf6116eee5bc811cc5d49c0e6ca03bc924ff1db688ba1773537041804442
Nov 03 21:33:34.911280 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:34.911224    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4466cc7-rbvjl" event={"ID":"ad574db0-89e0-4a72-84bf-bbd62702b536","Type":"ContainerStarted","Data":"e9debf6116eee5bc811cc5d49c0e6ca03bc924ff1db688ba1773537041804442"}
Nov 03 21:33:34.913358 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:34.913329    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc" event={"ID":"c43adf43-1729-4f0d-b6c7-3918c0fff604","Type":"ContainerStarted","Data":"3b74edf2f27c8fddc02faf0dc29fa92b58b753482abcc5f5fc886630bd90214e"}
Nov 03 21:33:34.913511 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:34.913384    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc" event={"ID":"c43adf43-1729-4f0d-b6c7-3918c0fff604","Type":"ContainerStarted","Data":"b11439c52b6c2fd009ae82c47f9319cc7bf6b61c49bb690c8d47f9a140a43f37"}
Nov 03 21:33:34.927857 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:34.927824    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:35.031727 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:35.031651    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc" podStartSLOduration=2.031632881 podStartE2EDuration="2.031632881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:33:35.027681631 +0000 UTC m=+1129.521603510" watchObservedRunningTime="2026-11-03 21:33:35.031632881 +0000 UTC m=+1129.525554757"
Nov 03 21:33:35.668262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:35.660047    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79b925eb-lpm74"]
Nov 03 21:33:35.700076 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:35.700047    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b925eb-lpm74"
Nov 03 21:33:35.781267 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:35.777843    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrrpn\" (UniqueName: \"kubernetes.io/projected/da076dbe-da94-4097-94e7-f133355e277d-kube-api-access-rrrpn\") pod \"maas-api-revoke-sub-79b925eb-lpm74\" (UID: \"da076dbe-da94-4097-94e7-f133355e277d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b925eb-lpm74"
Nov 03 21:33:35.781267 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:35.777987    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da076dbe-da94-4097-94e7-f133355e277d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79b925eb-lpm74\" (UID: \"da076dbe-da94-4097-94e7-f133355e277d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b925eb-lpm74"
Nov 03 21:33:35.781267 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:35.778056    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da076dbe-da94-4097-94e7-f133355e277d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79b925eb-lpm74\" (UID: \"da076dbe-da94-4097-94e7-f133355e277d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b925eb-lpm74"
Nov 03 21:33:35.794266 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:35.793566    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79b925eb-lpm74"]
Nov 03 21:33:35.879221 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:35.879187    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da076dbe-da94-4097-94e7-f133355e277d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79b925eb-lpm74\" (UID: \"da076dbe-da94-4097-94e7-f133355e277d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b925eb-lpm74"
Nov 03 21:33:35.879418 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:35.879343    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da076dbe-da94-4097-94e7-f133355e277d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79b925eb-lpm74\" (UID: \"da076dbe-da94-4097-94e7-f133355e277d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b925eb-lpm74"
Nov 03 21:33:35.879485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:35.879461    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrrpn\" (UniqueName: \"kubernetes.io/projected/da076dbe-da94-4097-94e7-f133355e277d-kube-api-access-rrrpn\") pod \"maas-api-revoke-sub-79b925eb-lpm74\" (UID: \"da076dbe-da94-4097-94e7-f133355e277d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b925eb-lpm74"
Nov 03 21:33:35.880307 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:35.880226    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da076dbe-da94-4097-94e7-f133355e277d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79b925eb-lpm74\" (UID: \"da076dbe-da94-4097-94e7-f133355e277d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b925eb-lpm74"
Nov 03 21:33:35.881755 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:35.881727    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da076dbe-da94-4097-94e7-f133355e277d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79b925eb-lpm74\" (UID: \"da076dbe-da94-4097-94e7-f133355e277d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b925eb-lpm74"
Nov 03 21:33:35.938755 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:35.938718    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 03 21:33:35.939220 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:35.938797    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc" podUID="c43adf43-1729-4f0d-b6c7-3918c0fff604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 03 21:33:35.941878 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:35.941857    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrrpn\" (UniqueName: \"kubernetes.io/projected/da076dbe-da94-4097-94e7-f133355e277d-kube-api-access-rrrpn\") pod \"maas-api-revoke-sub-79b925eb-lpm74\" (UID: \"da076dbe-da94-4097-94e7-f133355e277d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b925eb-lpm74"
Nov 03 21:33:35.950292 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:35.949612    5053 generic.go:356] "Generic (PLEG): container finished" podID="ad574db0-89e0-4a72-84bf-bbd62702b536" containerID="60f30c6f8f4b9b0d1e7ac782e7f2dc6fd263b226478f3814269fb3e01dd34a30" exitCode=0
Nov 03 21:33:35.950292 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:35.949902    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4466cc7-rbvjl" event={"ID":"ad574db0-89e0-4a72-84bf-bbd62702b536","Type":"ContainerDied","Data":"60f30c6f8f4b9b0d1e7ac782e7f2dc6fd263b226478f3814269fb3e01dd34a30"}
Nov 03 21:33:36.145566 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:36.145465    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b925eb-lpm74"
Nov 03 21:33:36.632328 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:36.626904    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79b925eb-lpm74"]
Nov 03 21:33:36.640196 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:33:36.640157    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda076dbe_da94_4097_94e7_f133355e277d.slice/crio-b1450fbdd6949d7b323b26fb48a4997e71a56b4b3dfb67cb27dd8f14ad2b2674 WatchSource:0}: Error finding container b1450fbdd6949d7b323b26fb48a4997e71a56b4b3dfb67cb27dd8f14ad2b2674: Status 404 returned error can't find the container with id b1450fbdd6949d7b323b26fb48a4997e71a56b4b3dfb67cb27dd8f14ad2b2674
Nov 03 21:33:36.656868 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:36.656834    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8"]
Nov 03 21:33:36.658903 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:36.658878    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8" podUID="bffc29d5-d3bb-429c-8772-7b672dd1f25f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:33:36.660934 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:36.660908    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7868f59688-cw6fs"]
Nov 03 21:33:36.933964 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:36.933883    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 03 21:33:36.934386 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:36.934341    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc" podUID="c43adf43-1729-4f0d-b6c7-3918c0fff604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 03 21:33:36.958291 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:36.958262    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b925eb-lpm74" event={"ID":"da076dbe-da94-4097-94e7-f133355e277d","Type":"ContainerStarted","Data":"b1450fbdd6949d7b323b26fb48a4997e71a56b4b3dfb67cb27dd8f14ad2b2674"}
Nov 03 21:33:37.164179 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:37.164155    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4466cc7-rbvjl"
Nov 03 21:33:37.198627 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:37.198606    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad574db0-89e0-4a72-84bf-bbd62702b536-kube-api-access-dfqdz\" (UniqueName: \"kubernetes.io/projected/ad574db0-89e0-4a72-84bf-bbd62702b536-kube-api-access-dfqdz\") pod \"ad574db0-89e0-4a72-84bf-bbd62702b536\" (UID: \"ad574db0-89e0-4a72-84bf-bbd62702b536\") "
Nov 03 21:33:37.198757 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:37.198694    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad574db0-89e0-4a72-84bf-bbd62702b536-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad574db0-89e0-4a72-84bf-bbd62702b536-maas-api-xks-ca\") pod \"ad574db0-89e0-4a72-84bf-bbd62702b536\" (UID: \"ad574db0-89e0-4a72-84bf-bbd62702b536\") "
Nov 03 21:33:37.198808 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:37.198762    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad574db0-89e0-4a72-84bf-bbd62702b536-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad574db0-89e0-4a72-84bf-bbd62702b536-maas-api-openshift-ca\") pod \"ad574db0-89e0-4a72-84bf-bbd62702b536\" (UID: \"ad574db0-89e0-4a72-84bf-bbd62702b536\") "
Nov 03 21:33:37.199104 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:37.199081    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad574db0-89e0-4a72-84bf-bbd62702b536-maas-api-openshift-ca" pod "ad574db0-89e0-4a72-84bf-bbd62702b536" (UID: "ad574db0-89e0-4a72-84bf-bbd62702b536"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:33:37.199397 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:37.199380    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad574db0-89e0-4a72-84bf-bbd62702b536-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:37.222509 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:37.220325    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad574db0-89e0-4a72-84bf-bbd62702b536-maas-api-xks-ca" pod "ad574db0-89e0-4a72-84bf-bbd62702b536" (UID: "ad574db0-89e0-4a72-84bf-bbd62702b536"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:33:37.236117 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:37.236074    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad574db0-89e0-4a72-84bf-bbd62702b536-kube-api-access-dfqdz" pod "ad574db0-89e0-4a72-84bf-bbd62702b536" (UID: "ad574db0-89e0-4a72-84bf-bbd62702b536"). InnerVolumeSpecName "kube-api-access-dfqdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:33:37.308267 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:37.300825    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfqdz\" (UniqueName: \"kubernetes.io/projected/ad574db0-89e0-4a72-84bf-bbd62702b536-kube-api-access-dfqdz\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:37.308267 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:37.300863    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad574db0-89e0-4a72-84bf-bbd62702b536-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:37.934655 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:37.932290    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 03 21:33:37.934655 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:37.932344    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc" podUID="c43adf43-1729-4f0d-b6c7-3918c0fff604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 03 21:33:37.965822 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:37.965792    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4466cc7-rbvjl"
Nov 03 21:33:37.966257 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:37.965787    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4466cc7-rbvjl" event={"ID":"ad574db0-89e0-4a72-84bf-bbd62702b536","Type":"ContainerDied","Data":"e9debf6116eee5bc811cc5d49c0e6ca03bc924ff1db688ba1773537041804442"}
Nov 03 21:33:37.966257 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:37.965944    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9debf6116eee5bc811cc5d49c0e6ca03bc924ff1db688ba1773537041804442"
Nov 03 21:33:37.967639 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:37.967596    5053 generic.go:356] "Generic (PLEG): container finished" podID="da076dbe-da94-4097-94e7-f133355e277d" containerID="218f3678c71e2116d6cd5e320ae1ebfe698f1dbdadc9e0483e62d462fbea5300" exitCode=0
Nov 03 21:33:37.967767 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:37.967630    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b925eb-lpm74" event={"ID":"da076dbe-da94-4097-94e7-f133355e277d","Type":"ContainerDied","Data":"218f3678c71e2116d6cd5e320ae1ebfe698f1dbdadc9e0483e62d462fbea5300"}
Nov 03 21:33:38.927291 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:38.927248    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 03 21:33:38.927515 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:38.927315    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc" podUID="c43adf43-1729-4f0d-b6c7-3918c0fff604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 03 21:33:39.164096 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:39.164067    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b925eb-lpm74"
Nov 03 21:33:39.264540 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:39.264500    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da076dbe-da94-4097-94e7-f133355e277d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da076dbe-da94-4097-94e7-f133355e277d-maas-api-xks-ca\") pod \"da076dbe-da94-4097-94e7-f133355e277d\" (UID: \"da076dbe-da94-4097-94e7-f133355e277d\") "
Nov 03 21:33:39.264753 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:39.264707    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da076dbe-da94-4097-94e7-f133355e277d-kube-api-access-rrrpn\" (UniqueName: \"kubernetes.io/projected/da076dbe-da94-4097-94e7-f133355e277d-kube-api-access-rrrpn\") pod \"da076dbe-da94-4097-94e7-f133355e277d\" (UID: \"da076dbe-da94-4097-94e7-f133355e277d\") "
Nov 03 21:33:39.264822 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:39.264798    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da076dbe-da94-4097-94e7-f133355e277d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da076dbe-da94-4097-94e7-f133355e277d-maas-api-openshift-ca\") pod \"da076dbe-da94-4097-94e7-f133355e277d\" (UID: \"da076dbe-da94-4097-94e7-f133355e277d\") "
Nov 03 21:33:39.265097 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:39.265069    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da076dbe-da94-4097-94e7-f133355e277d-maas-api-openshift-ca" pod "da076dbe-da94-4097-94e7-f133355e277d" (UID: "da076dbe-da94-4097-94e7-f133355e277d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:33:39.265293 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:39.265229    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da076dbe-da94-4097-94e7-f133355e277d-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:39.266865 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:39.266841    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da076dbe-da94-4097-94e7-f133355e277d-maas-api-xks-ca" pod "da076dbe-da94-4097-94e7-f133355e277d" (UID: "da076dbe-da94-4097-94e7-f133355e277d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:33:39.266962 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:39.266929    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da076dbe-da94-4097-94e7-f133355e277d-kube-api-access-rrrpn" pod "da076dbe-da94-4097-94e7-f133355e277d" (UID: "da076dbe-da94-4097-94e7-f133355e277d"). InnerVolumeSpecName "kube-api-access-rrrpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:33:39.366947 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:39.366902    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da076dbe-da94-4097-94e7-f133355e277d-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:39.366947 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:39.366938    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrrpn\" (UniqueName: \"kubernetes.io/projected/da076dbe-da94-4097-94e7-f133355e277d-kube-api-access-rrrpn\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:39.927360 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:39.927253    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 03 21:33:39.927360 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:39.927325    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc" podUID="c43adf43-1729-4f0d-b6c7-3918c0fff604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 03 21:33:39.983543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:39.983520    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b925eb-lpm74"
Nov 03 21:33:39.983691 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:39.983514    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b925eb-lpm74" event={"ID":"da076dbe-da94-4097-94e7-f133355e277d","Type":"ContainerDied","Data":"b1450fbdd6949d7b323b26fb48a4997e71a56b4b3dfb67cb27dd8f14ad2b2674"}
Nov 03 21:33:39.983691 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:39.983641    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1450fbdd6949d7b323b26fb48a4997e71a56b4b3dfb67cb27dd8f14ad2b2674"
Nov 03 21:33:40.927421 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:40.927384    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 03 21:33:40.927933 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:40.927454    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc" podUID="c43adf43-1729-4f0d-b6c7-3918c0fff604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 03 21:33:40.928305 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:33:40.928270    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:33:41.657419 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:41.657372    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8" podUID="bffc29d5-d3bb-429c-8772-7b672dd1f25f" containerName="payload-processing" containerID="cri-o://c20fa0b8e1e0cde2c4e0695acf38f936367dc1eae836dafea79d23b2e089016a" gracePeriod=25
Nov 03 21:33:41.661421 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:41.661375    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7868f59688-cw6fs" podUID="f53fe56d-d4f6-4754-a27b-87f71b5895db" containerName="payload-processing" containerID="cri-o://fdc801a59da4723f34101bb058bab315ff0a7c4c0c0815d1643ae23ef3482e02" gracePeriod=25
Nov 03 21:33:41.970334 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:41.968501    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8"
Nov 03 21:33:41.999421 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:41.999199    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:41.999742 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:41.999718    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:42.001956 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.001832    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:33:42.002329 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.002299    5053 generic.go:356] "Generic (PLEG): container finished" podID="bffc29d5-d3bb-429c-8772-7b672dd1f25f" containerID="c20fa0b8e1e0cde2c4e0695acf38f936367dc1eae836dafea79d23b2e089016a" exitCode=0
Nov 03 21:33:42.002329 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.002319    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8"
Nov 03 21:33:42.002461 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.002349    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8" event={"ID":"bffc29d5-d3bb-429c-8772-7b672dd1f25f","Type":"ContainerDied","Data":"c20fa0b8e1e0cde2c4e0695acf38f936367dc1eae836dafea79d23b2e089016a"}
Nov 03 21:33:42.002461 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.002391    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8" event={"ID":"bffc29d5-d3bb-429c-8772-7b672dd1f25f","Type":"ContainerDied","Data":"62c879371234f4cacabca20fffe203dbeb8a4a1289dcc71b4aaad0c2c0eb1959"}
Nov 03 21:33:42.002461 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.002416    5053 scope.go:122] "RemoveContainer" containerID="c20fa0b8e1e0cde2c4e0695acf38f936367dc1eae836dafea79d23b2e089016a"
Nov 03 21:33:42.004338 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.004159    5053 generic.go:356] "Generic (PLEG): container finished" podID="f53fe56d-d4f6-4754-a27b-87f71b5895db" containerID="fdc801a59da4723f34101bb058bab315ff0a7c4c0c0815d1643ae23ef3482e02" exitCode=0
Nov 03 21:33:42.004585 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.004563    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7868f59688-cw6fs" event={"ID":"f53fe56d-d4f6-4754-a27b-87f71b5895db","Type":"ContainerDied","Data":"fdc801a59da4723f34101bb058bab315ff0a7c4c0c0815d1643ae23ef3482e02"}
Nov 03 21:33:42.004655 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.004611    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7868f59688-cw6fs" event={"ID":"f53fe56d-d4f6-4754-a27b-87f71b5895db","Type":"ContainerDied","Data":"a463220381f4c846ecfcc0d416f4fd967ae08026f289509113bd44d49a18baa0"}
Nov 03 21:33:42.004655 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.004639    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a463220381f4c846ecfcc0d416f4fd967ae08026f289509113bd44d49a18baa0"
Nov 03 21:33:42.005163 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.005144    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bffc29d5-d3bb-429c-8772-7b672dd1f25f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bffc29d5-d3bb-429c-8772-7b672dd1f25f-plugins-config-volume\") pod \"bffc29d5-d3bb-429c-8772-7b672dd1f25f\" (UID: \"bffc29d5-d3bb-429c-8772-7b672dd1f25f\") "
Nov 03 21:33:42.005316 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.005298    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bffc29d5-d3bb-429c-8772-7b672dd1f25f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bffc29d5-d3bb-429c-8772-7b672dd1f25f-routing-overlay-volume\") pod \"bffc29d5-d3bb-429c-8772-7b672dd1f25f\" (UID: \"bffc29d5-d3bb-429c-8772-7b672dd1f25f\") "
Nov 03 21:33:42.005428 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.005413    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bffc29d5-d3bb-429c-8772-7b672dd1f25f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bffc29d5-d3bb-429c-8772-7b672dd1f25f-provider-credentials\") pod \"bffc29d5-d3bb-429c-8772-7b672dd1f25f\" (UID: \"bffc29d5-d3bb-429c-8772-7b672dd1f25f\") "
Nov 03 21:33:42.005485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.005476    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bffc29d5-d3bb-429c-8772-7b672dd1f25f-kube-api-access-57pmx\" (UniqueName: \"kubernetes.io/projected/bffc29d5-d3bb-429c-8772-7b672dd1f25f-kube-api-access-57pmx\") pod \"bffc29d5-d3bb-429c-8772-7b672dd1f25f\" (UID: \"bffc29d5-d3bb-429c-8772-7b672dd1f25f\") "
Nov 03 21:33:42.005702 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.005625    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bffc29d5-d3bb-429c-8772-7b672dd1f25f-plugins-config-volume" pod "bffc29d5-d3bb-429c-8772-7b672dd1f25f" (UID: "bffc29d5-d3bb-429c-8772-7b672dd1f25f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:33:42.005873 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.005852    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bffc29d5-d3bb-429c-8772-7b672dd1f25f-routing-overlay-volume" pod "bffc29d5-d3bb-429c-8772-7b672dd1f25f" (UID: "bffc29d5-d3bb-429c-8772-7b672dd1f25f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:33:42.006008 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.005986    5053 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bffc29d5-d3bb-429c-8772-7b672dd1f25f-routing-overlay-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:42.006225 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.006012    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bffc29d5-d3bb-429c-8772-7b672dd1f25f-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:42.007994 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.007970    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bffc29d5-d3bb-429c-8772-7b672dd1f25f-kube-api-access-57pmx" pod "bffc29d5-d3bb-429c-8772-7b672dd1f25f" (UID: "bffc29d5-d3bb-429c-8772-7b672dd1f25f"). InnerVolumeSpecName "kube-api-access-57pmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:33:42.008226 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.008204    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bffc29d5-d3bb-429c-8772-7b672dd1f25f-provider-credentials" pod "bffc29d5-d3bb-429c-8772-7b672dd1f25f" (UID: "bffc29d5-d3bb-429c-8772-7b672dd1f25f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:33:42.014226 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.014207    5053 scope.go:122] "RemoveContainer" containerID="c20fa0b8e1e0cde2c4e0695acf38f936367dc1eae836dafea79d23b2e089016a"
Nov 03 21:33:42.014504 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:33:42.014482    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c20fa0b8e1e0cde2c4e0695acf38f936367dc1eae836dafea79d23b2e089016a\": container with ID starting with c20fa0b8e1e0cde2c4e0695acf38f936367dc1eae836dafea79d23b2e089016a not found: ID does not exist" containerID="c20fa0b8e1e0cde2c4e0695acf38f936367dc1eae836dafea79d23b2e089016a"
Nov 03 21:33:42.014592 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.014516    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c20fa0b8e1e0cde2c4e0695acf38f936367dc1eae836dafea79d23b2e089016a"} err="failed to get container status \"c20fa0b8e1e0cde2c4e0695acf38f936367dc1eae836dafea79d23b2e089016a\": rpc error: code = NotFound desc = could not find container \"c20fa0b8e1e0cde2c4e0695acf38f936367dc1eae836dafea79d23b2e089016a\": container with ID starting with c20fa0b8e1e0cde2c4e0695acf38f936367dc1eae836dafea79d23b2e089016a not found: ID does not exist"
Nov 03 21:33:42.016787 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.016768    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7868f59688-cw6fs"
Nov 03 21:33:42.106972 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.106936    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f53fe56d-d4f6-4754-a27b-87f71b5895db-kube-api-access-4nwps\" (UniqueName: \"kubernetes.io/projected/f53fe56d-d4f6-4754-a27b-87f71b5895db-kube-api-access-4nwps\") pod \"f53fe56d-d4f6-4754-a27b-87f71b5895db\" (UID: \"f53fe56d-d4f6-4754-a27b-87f71b5895db\") "
Nov 03 21:33:42.107159 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.106988    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f53fe56d-d4f6-4754-a27b-87f71b5895db-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f53fe56d-d4f6-4754-a27b-87f71b5895db-provider-credentials\") pod \"f53fe56d-d4f6-4754-a27b-87f71b5895db\" (UID: \"f53fe56d-d4f6-4754-a27b-87f71b5895db\") "
Nov 03 21:33:42.107159 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.107071    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f53fe56d-d4f6-4754-a27b-87f71b5895db-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f53fe56d-d4f6-4754-a27b-87f71b5895db-routing-overlay-volume\") pod \"f53fe56d-d4f6-4754-a27b-87f71b5895db\" (UID: \"f53fe56d-d4f6-4754-a27b-87f71b5895db\") "
Nov 03 21:33:42.107345 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.107168    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f53fe56d-d4f6-4754-a27b-87f71b5895db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f53fe56d-d4f6-4754-a27b-87f71b5895db-plugins-config-volume\") pod \"f53fe56d-d4f6-4754-a27b-87f71b5895db\" (UID: \"f53fe56d-d4f6-4754-a27b-87f71b5895db\") "
Nov 03 21:33:42.108688 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.107684    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f53fe56d-d4f6-4754-a27b-87f71b5895db-routing-overlay-volume" pod "f53fe56d-d4f6-4754-a27b-87f71b5895db" (UID: "f53fe56d-d4f6-4754-a27b-87f71b5895db"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:33:42.108688 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.107998    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f53fe56d-d4f6-4754-a27b-87f71b5895db-plugins-config-volume" pod "f53fe56d-d4f6-4754-a27b-87f71b5895db" (UID: "f53fe56d-d4f6-4754-a27b-87f71b5895db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:33:42.108688 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.108409    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f53fe56d-d4f6-4754-a27b-87f71b5895db-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:42.108688 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.108433    5053 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f53fe56d-d4f6-4754-a27b-87f71b5895db-routing-overlay-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:42.108688 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.108448    5053 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bffc29d5-d3bb-429c-8772-7b672dd1f25f-provider-credentials\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:42.108688 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.108461    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57pmx\" (UniqueName: \"kubernetes.io/projected/bffc29d5-d3bb-429c-8772-7b672dd1f25f-kube-api-access-57pmx\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:42.110106 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.110078    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53fe56d-d4f6-4754-a27b-87f71b5895db-kube-api-access-4nwps" pod "f53fe56d-d4f6-4754-a27b-87f71b5895db" (UID: "f53fe56d-d4f6-4754-a27b-87f71b5895db"). InnerVolumeSpecName "kube-api-access-4nwps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:33:42.110599 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.110575    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53fe56d-d4f6-4754-a27b-87f71b5895db-provider-credentials" pod "f53fe56d-d4f6-4754-a27b-87f71b5895db" (UID: "f53fe56d-d4f6-4754-a27b-87f71b5895db"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:33:42.209840 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.209809    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nwps\" (UniqueName: \"kubernetes.io/projected/f53fe56d-d4f6-4754-a27b-87f71b5895db-kube-api-access-4nwps\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:42.209840 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.209839    5053 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f53fe56d-d4f6-4754-a27b-87f71b5895db-provider-credentials\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:42.528202 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.528120    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8"]
Nov 03 21:33:42.533456 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:42.533421    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6575949dfc-ntpn8"]
Nov 03 21:33:43.013400 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:43.013370    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7868f59688-cw6fs"
Nov 03 21:33:43.177505 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:43.177469    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7868f59688-cw6fs"]
Nov 03 21:33:43.181117 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:43.181093    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7868f59688-cw6fs"]
Nov 03 21:33:43.669219 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:43.669181    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bffc29d5-d3bb-429c-8772-7b672dd1f25f" path="/var/lib/kubelet/pods/bffc29d5-d3bb-429c-8772-7b672dd1f25f/volumes"
Nov 03 21:33:43.669661 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:43.669647    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f53fe56d-d4f6-4754-a27b-87f71b5895db" path="/var/lib/kubelet/pods/f53fe56d-d4f6-4754-a27b-87f71b5895db/volumes"
Nov 03 21:33:44.614833 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.614805    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d6a96e0-ndbbl"]
Nov 03 21:33:44.615423 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.615409    5053 cpu_manager.go:414] "containerMap: removing container" podUID="bffc29d5-d3bb-429c-8772-7b672dd1f25f" containerName="payload-processing"
Nov 03 21:33:44.615478 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.615424    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bffc29d5-d3bb-429c-8772-7b672dd1f25f" containerName="payload-processing"
Nov 03 21:33:44.615478 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.615439    5053 cpu_manager.go:414] "containerMap: removing container" podUID="ad574db0-89e0-4a72-84bf-bbd62702b536" containerName="revoke-keys"
Nov 03 21:33:44.615478 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.615444    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad574db0-89e0-4a72-84bf-bbd62702b536" containerName="revoke-keys"
Nov 03 21:33:44.615478 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.615455    5053 cpu_manager.go:414] "containerMap: removing container" podUID="f53fe56d-d4f6-4754-a27b-87f71b5895db" containerName="payload-processing"
Nov 03 21:33:44.615478 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.615459    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f53fe56d-d4f6-4754-a27b-87f71b5895db" containerName="payload-processing"
Nov 03 21:33:44.615620 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.615494    5053 cpu_manager.go:414] "containerMap: removing container" podUID="da076dbe-da94-4097-94e7-f133355e277d" containerName="revoke-keys"
Nov 03 21:33:44.615620 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.615498    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da076dbe-da94-4097-94e7-f133355e277d" containerName="revoke-keys"
Nov 03 21:33:44.615620 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.615583    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="f53fe56d-d4f6-4754-a27b-87f71b5895db" containerName="payload-processing"
Nov 03 21:33:44.615620 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.615598    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="da076dbe-da94-4097-94e7-f133355e277d" containerName="revoke-keys"
Nov 03 21:33:44.615620 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.615604    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="bffc29d5-d3bb-429c-8772-7b672dd1f25f" containerName="payload-processing"
Nov 03 21:33:44.615620 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.615609    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad574db0-89e0-4a72-84bf-bbd62702b536" containerName="revoke-keys"
Nov 03 21:33:44.620547 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.620529    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a96e0-ndbbl"
Nov 03 21:33:44.674966 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.674933    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d6a96e0-ndbbl"]
Nov 03 21:33:44.720383 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.720325    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:33:44.720383 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.720359    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:33:44.720383 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.720370    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:33:44.720684 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.720642    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:33:44.741399 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.741373    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4swxg\" (UniqueName: \"kubernetes.io/projected/e72ac212-56b7-4d76-a32f-6ee939a927b5-kube-api-access-4swxg\") pod \"maas-api-revoke-sub-8d6a96e0-ndbbl\" (UID: \"e72ac212-56b7-4d76-a32f-6ee939a927b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a96e0-ndbbl"
Nov 03 21:33:44.741541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.741456    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72ac212-56b7-4d76-a32f-6ee939a927b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d6a96e0-ndbbl\" (UID: \"e72ac212-56b7-4d76-a32f-6ee939a927b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a96e0-ndbbl"
Nov 03 21:33:44.741637 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.741616    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72ac212-56b7-4d76-a32f-6ee939a927b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d6a96e0-ndbbl\" (UID: \"e72ac212-56b7-4d76-a32f-6ee939a927b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a96e0-ndbbl"
Nov 03 21:33:44.843371 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.843338    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72ac212-56b7-4d76-a32f-6ee939a927b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d6a96e0-ndbbl\" (UID: \"e72ac212-56b7-4d76-a32f-6ee939a927b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a96e0-ndbbl"
Nov 03 21:33:44.843562 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.843450    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4swxg\" (UniqueName: \"kubernetes.io/projected/e72ac212-56b7-4d76-a32f-6ee939a927b5-kube-api-access-4swxg\") pod \"maas-api-revoke-sub-8d6a96e0-ndbbl\" (UID: \"e72ac212-56b7-4d76-a32f-6ee939a927b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a96e0-ndbbl"
Nov 03 21:33:44.843562 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.843524    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72ac212-56b7-4d76-a32f-6ee939a927b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d6a96e0-ndbbl\" (UID: \"e72ac212-56b7-4d76-a32f-6ee939a927b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a96e0-ndbbl"
Nov 03 21:33:44.844147 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.844117    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72ac212-56b7-4d76-a32f-6ee939a927b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d6a96e0-ndbbl\" (UID: \"e72ac212-56b7-4d76-a32f-6ee939a927b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a96e0-ndbbl"
Nov 03 21:33:44.845842 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.845820    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72ac212-56b7-4d76-a32f-6ee939a927b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d6a96e0-ndbbl\" (UID: \"e72ac212-56b7-4d76-a32f-6ee939a927b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a96e0-ndbbl"
Nov 03 21:33:44.899864 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.899801    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4swxg\" (UniqueName: \"kubernetes.io/projected/e72ac212-56b7-4d76-a32f-6ee939a927b5-kube-api-access-4swxg\") pod \"maas-api-revoke-sub-8d6a96e0-ndbbl\" (UID: \"e72ac212-56b7-4d76-a32f-6ee939a927b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a96e0-ndbbl"
Nov 03 21:33:44.934123 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:44.934098    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a96e0-ndbbl"
Nov 03 21:33:45.315469 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:45.315435    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d6a96e0-ndbbl"]
Nov 03 21:33:45.316028 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:33:45.316007    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode72ac212_56b7_4d76_a32f_6ee939a927b5.slice/crio-0ada9532051cf5f4ff6cddf559a25067f1a85bc365876fd599efdb0438daeaa0 WatchSource:0}: Error finding container 0ada9532051cf5f4ff6cddf559a25067f1a85bc365876fd599efdb0438daeaa0: Status 404 returned error can't find the container with id 0ada9532051cf5f4ff6cddf559a25067f1a85bc365876fd599efdb0438daeaa0
Nov 03 21:33:46.039715 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:46.039605    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a96e0-ndbbl" event={"ID":"e72ac212-56b7-4d76-a32f-6ee939a927b5","Type":"ContainerStarted","Data":"f0c40b2358feab6b720c07e6ee513cb399677a1a39898041adf158f12d0abc5a"}
Nov 03 21:33:46.039715 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:46.039679    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a96e0-ndbbl" event={"ID":"e72ac212-56b7-4d76-a32f-6ee939a927b5","Type":"ContainerStarted","Data":"0ada9532051cf5f4ff6cddf559a25067f1a85bc365876fd599efdb0438daeaa0"}
Nov 03 21:33:46.156066 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:46.156016    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a96e0-ndbbl" podStartSLOduration=2.15599976 podStartE2EDuration="2.15599976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:33:46.152137765 +0000 UTC m=+1140.646059641" watchObservedRunningTime="2026-11-03 21:33:46.15599976 +0000 UTC m=+1140.649921636"
Nov 03 21:33:47.049191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:47.049152    5053 generic.go:356] "Generic (PLEG): container finished" podID="e72ac212-56b7-4d76-a32f-6ee939a927b5" containerID="f0c40b2358feab6b720c07e6ee513cb399677a1a39898041adf158f12d0abc5a" exitCode=0
Nov 03 21:33:47.049627 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:47.049299    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a96e0-ndbbl" event={"ID":"e72ac212-56b7-4d76-a32f-6ee939a927b5","Type":"ContainerDied","Data":"f0c40b2358feab6b720c07e6ee513cb399677a1a39898041adf158f12d0abc5a"}
Nov 03 21:33:48.212513 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:48.212484    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a96e0-ndbbl"
Nov 03 21:33:48.289475 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:48.289434    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e72ac212-56b7-4d76-a32f-6ee939a927b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72ac212-56b7-4d76-a32f-6ee939a927b5-maas-api-openshift-ca\") pod \"e72ac212-56b7-4d76-a32f-6ee939a927b5\" (UID: \"e72ac212-56b7-4d76-a32f-6ee939a927b5\") "
Nov 03 21:33:48.289475 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:48.289489    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e72ac212-56b7-4d76-a32f-6ee939a927b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72ac212-56b7-4d76-a32f-6ee939a927b5-maas-api-xks-ca\") pod \"e72ac212-56b7-4d76-a32f-6ee939a927b5\" (UID: \"e72ac212-56b7-4d76-a32f-6ee939a927b5\") "
Nov 03 21:33:48.289727 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:48.289519    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e72ac212-56b7-4d76-a32f-6ee939a927b5-kube-api-access-4swxg\" (UniqueName: \"kubernetes.io/projected/e72ac212-56b7-4d76-a32f-6ee939a927b5-kube-api-access-4swxg\") pod \"e72ac212-56b7-4d76-a32f-6ee939a927b5\" (UID: \"e72ac212-56b7-4d76-a32f-6ee939a927b5\") "
Nov 03 21:33:48.289842 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:48.289817    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e72ac212-56b7-4d76-a32f-6ee939a927b5-maas-api-openshift-ca" pod "e72ac212-56b7-4d76-a32f-6ee939a927b5" (UID: "e72ac212-56b7-4d76-a32f-6ee939a927b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:33:48.289959 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:48.289946    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72ac212-56b7-4d76-a32f-6ee939a927b5-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:48.291708 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:48.291686    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e72ac212-56b7-4d76-a32f-6ee939a927b5-maas-api-xks-ca" pod "e72ac212-56b7-4d76-a32f-6ee939a927b5" (UID: "e72ac212-56b7-4d76-a32f-6ee939a927b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:33:48.291824 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:48.291783    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e72ac212-56b7-4d76-a32f-6ee939a927b5-kube-api-access-4swxg" pod "e72ac212-56b7-4d76-a32f-6ee939a927b5" (UID: "e72ac212-56b7-4d76-a32f-6ee939a927b5"). InnerVolumeSpecName "kube-api-access-4swxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:33:48.391119 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:48.391041    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72ac212-56b7-4d76-a32f-6ee939a927b5-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:48.391119 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:48.391069    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4swxg\" (UniqueName: \"kubernetes.io/projected/e72ac212-56b7-4d76-a32f-6ee939a927b5-kube-api-access-4swxg\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:49.064605 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:49.064577    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a96e0-ndbbl"
Nov 03 21:33:49.064785 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:49.064603    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a96e0-ndbbl" event={"ID":"e72ac212-56b7-4d76-a32f-6ee939a927b5","Type":"ContainerDied","Data":"0ada9532051cf5f4ff6cddf559a25067f1a85bc365876fd599efdb0438daeaa0"}
Nov 03 21:33:49.064785 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:49.064649    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ada9532051cf5f4ff6cddf559a25067f1a85bc365876fd599efdb0438daeaa0"
Nov 03 21:33:51.998258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:51.994433    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-142a67da-vh2r8"]
Nov 03 21:33:51.998258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:51.995566    5053 cpu_manager.go:414] "containerMap: removing container" podUID="e72ac212-56b7-4d76-a32f-6ee939a927b5" containerName="revoke-keys"
Nov 03 21:33:51.998258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:51.995586    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e72ac212-56b7-4d76-a32f-6ee939a927b5" containerName="revoke-keys"
Nov 03 21:33:51.998258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:51.995795    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="e72ac212-56b7-4d76-a32f-6ee939a927b5" containerName="revoke-keys"
Nov 03 21:33:52.022265 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:52.019185    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-142a67da-vh2r8"
Nov 03 21:33:52.048262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:52.032544    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-142a67da-vh2r8"]
Nov 03 21:33:52.118258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:52.100608    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:33:52.174267 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:52.164533    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:33:52.186367 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:52.186334    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:33:52.192313 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:52.187469    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:33:52.196265 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:52.188088    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26ce17c8-b884-4902-b42a-b83005710f38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-142a67da-vh2r8\" (UID: \"26ce17c8-b884-4902-b42a-b83005710f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-142a67da-vh2r8"
Nov 03 21:33:52.221527 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:52.221494    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtwhp\" (UniqueName: \"kubernetes.io/projected/26ce17c8-b884-4902-b42a-b83005710f38-kube-api-access-xtwhp\") pod \"maas-api-revoke-sub-142a67da-vh2r8\" (UID: \"26ce17c8-b884-4902-b42a-b83005710f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-142a67da-vh2r8"
Nov 03 21:33:52.229254 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:52.221921    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26ce17c8-b884-4902-b42a-b83005710f38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-142a67da-vh2r8\" (UID: \"26ce17c8-b884-4902-b42a-b83005710f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-142a67da-vh2r8"
Nov 03 21:33:52.374325 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:52.349537    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26ce17c8-b884-4902-b42a-b83005710f38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-142a67da-vh2r8\" (UID: \"26ce17c8-b884-4902-b42a-b83005710f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-142a67da-vh2r8"
Nov 03 21:33:52.374325 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:52.349616    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26ce17c8-b884-4902-b42a-b83005710f38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-142a67da-vh2r8\" (UID: \"26ce17c8-b884-4902-b42a-b83005710f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-142a67da-vh2r8"
Nov 03 21:33:52.374325 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:52.349708    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtwhp\" (UniqueName: \"kubernetes.io/projected/26ce17c8-b884-4902-b42a-b83005710f38-kube-api-access-xtwhp\") pod \"maas-api-revoke-sub-142a67da-vh2r8\" (UID: \"26ce17c8-b884-4902-b42a-b83005710f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-142a67da-vh2r8"
Nov 03 21:33:52.399298 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:52.377057    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26ce17c8-b884-4902-b42a-b83005710f38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-142a67da-vh2r8\" (UID: \"26ce17c8-b884-4902-b42a-b83005710f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-142a67da-vh2r8"
Nov 03 21:33:52.471409 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:52.471337    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtwhp\" (UniqueName: \"kubernetes.io/projected/26ce17c8-b884-4902-b42a-b83005710f38-kube-api-access-xtwhp\") pod \"maas-api-revoke-sub-142a67da-vh2r8\" (UID: \"26ce17c8-b884-4902-b42a-b83005710f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-142a67da-vh2r8"
Nov 03 21:33:52.472097 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:52.472037    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26ce17c8-b884-4902-b42a-b83005710f38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-142a67da-vh2r8\" (UID: \"26ce17c8-b884-4902-b42a-b83005710f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-142a67da-vh2r8"
Nov 03 21:33:52.507806 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:52.507484    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-142a67da-vh2r8"
Nov 03 21:33:53.347263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:53.343367    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8jgdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:33:53.347263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:53.343437    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx" podUID="909e67ce-5511-4f93-91eb-17f6a87acd84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:53.447592 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:53.447539    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n" podUID="512b3ab1-45ba-4f96-9799-b129b25fca42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:53.630460 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:53.630412    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6" podUID="4803807f-9312-40ac-badb-00c00d3f3bd3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 21:33:53.902336 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:53.895422    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx" podUID="a6c0d3aa-18c2-465b-846b-b82968acc780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 03 21:33:53.939264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:53.913951    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd" podUID="82077ea7-1e05-4b6e-806b-a2d2d2a64c4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 21:33:54.062919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:54.020058    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8" podUID="43404f1c-c87c-4325-ab97-bffb6ea81e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:54.114381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:54.106153    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-142a67da-vh2r8"]
Nov 03 21:33:54.308780 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:54.308434    5053 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx" podUID="a6c0d3aa-18c2-465b-846b-b82968acc780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:54.495578 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:54.493335    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n" podUID="512b3ab1-45ba-4f96-9799-b129b25fca42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:54.495578 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:54.494089    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:33:54.632262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:54.631948    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6" podUID="4803807f-9312-40ac-badb-00c00d3f3bd3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:54.682416 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:54.677909    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:33:54.792714 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:54.709730    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:33:54.792714 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:54.753774    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:33:54.964294 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:54.916425    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx" podUID="a6c0d3aa-18c2-465b-846b-b82968acc780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:54.964294 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:54.917053    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:33:54.977274 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:54.969348    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd" podUID="82077ea7-1e05-4b6e-806b-a2d2d2a64c4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:54.977274 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:54.972391    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:55.491380 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:55.460543    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-142a67da-vh2r8" event={"ID":"26ce17c8-b884-4902-b42a-b83005710f38","Type":"ContainerStarted","Data":"9a7cad4d0680bf780fb4a5a8f3e5c2ad04d024809630f57f0951771ac1dcc3a5"}
Nov 03 21:33:55.972084 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:55.972029    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd" podUID="82077ea7-1e05-4b6e-806b-a2d2d2a64c4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:55.979103 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:55.979070    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n" podUID="512b3ab1-45ba-4f96-9799-b129b25fca42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 03 21:33:55.983665 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:55.983630    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6" podUID="4803807f-9312-40ac-badb-00c00d3f3bd3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:56.028584 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:55.988893    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8" podUID="43404f1c-c87c-4325-ab97-bffb6ea81e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:56.037035 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:56.036524    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:33:56.073315 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:56.071830    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:33:56.477956 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:56.477915    5053 generic.go:356] "Generic (PLEG): container finished" podID="26ce17c8-b884-4902-b42a-b83005710f38" containerID="39c78fda274361b10fee3e809efcfad9c7112c440b552d30a2876f80c8a2fb9c" exitCode=0
Nov 03 21:33:56.478342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:56.478307    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-142a67da-vh2r8" event={"ID":"26ce17c8-b884-4902-b42a-b83005710f38","Type":"ContainerDied","Data":"39c78fda274361b10fee3e809efcfad9c7112c440b552d30a2876f80c8a2fb9c"}
Nov 03 21:33:57.653356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:57.653330    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-142a67da-vh2r8"
Nov 03 21:33:57.710451 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:57.710419    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26ce17c8-b884-4902-b42a-b83005710f38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26ce17c8-b884-4902-b42a-b83005710f38-maas-api-xks-ca\") pod \"26ce17c8-b884-4902-b42a-b83005710f38\" (UID: \"26ce17c8-b884-4902-b42a-b83005710f38\") "
Nov 03 21:33:57.710638 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:57.710583    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26ce17c8-b884-4902-b42a-b83005710f38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26ce17c8-b884-4902-b42a-b83005710f38-maas-api-openshift-ca\") pod \"26ce17c8-b884-4902-b42a-b83005710f38\" (UID: \"26ce17c8-b884-4902-b42a-b83005710f38\") "
Nov 03 21:33:57.710704 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:57.710670    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ce17c8-b884-4902-b42a-b83005710f38-kube-api-access-xtwhp\" (UniqueName: \"kubernetes.io/projected/26ce17c8-b884-4902-b42a-b83005710f38-kube-api-access-xtwhp\") pod \"26ce17c8-b884-4902-b42a-b83005710f38\" (UID: \"26ce17c8-b884-4902-b42a-b83005710f38\") "
Nov 03 21:33:57.710983 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:57.710949    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ce17c8-b884-4902-b42a-b83005710f38-maas-api-openshift-ca" pod "26ce17c8-b884-4902-b42a-b83005710f38" (UID: "26ce17c8-b884-4902-b42a-b83005710f38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:33:57.711270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:57.711251    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26ce17c8-b884-4902-b42a-b83005710f38-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:57.713256 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:57.713198    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26ce17c8-b884-4902-b42a-b83005710f38-maas-api-xks-ca" pod "26ce17c8-b884-4902-b42a-b83005710f38" (UID: "26ce17c8-b884-4902-b42a-b83005710f38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:33:57.713371 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:57.713296    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ce17c8-b884-4902-b42a-b83005710f38-kube-api-access-xtwhp" pod "26ce17c8-b884-4902-b42a-b83005710f38" (UID: "26ce17c8-b884-4902-b42a-b83005710f38"). InnerVolumeSpecName "kube-api-access-xtwhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:33:57.812872 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:57.812790    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26ce17c8-b884-4902-b42a-b83005710f38-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:57.812872 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:57.812819    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtwhp\" (UniqueName: \"kubernetes.io/projected/26ce17c8-b884-4902-b42a-b83005710f38-kube-api-access-xtwhp\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:33:58.493008 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:58.492979    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-142a67da-vh2r8"
Nov 03 21:33:58.493008 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:58.492998    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-142a67da-vh2r8" event={"ID":"26ce17c8-b884-4902-b42a-b83005710f38","Type":"ContainerDied","Data":"9a7cad4d0680bf780fb4a5a8f3e5c2ad04d024809630f57f0951771ac1dcc3a5"}
Nov 03 21:33:58.493248 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:33:58.493067    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a7cad4d0680bf780fb4a5a8f3e5c2ad04d024809630f57f0951771ac1dcc3a5"
Nov 03 21:34:01.343720 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:01.343684    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c39782a-rvqsn"]
Nov 03 21:34:01.344311 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:01.344296    5053 cpu_manager.go:414] "containerMap: removing container" podUID="26ce17c8-b884-4902-b42a-b83005710f38" containerName="revoke-keys"
Nov 03 21:34:01.344370 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:01.344312    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ce17c8-b884-4902-b42a-b83005710f38" containerName="revoke-keys"
Nov 03 21:34:01.344441 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:01.344432    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ce17c8-b884-4902-b42a-b83005710f38" containerName="revoke-keys"
Nov 03 21:34:01.347980 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:01.347960    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c39782a-rvqsn"
Nov 03 21:34:01.404420 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:01.404388    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c39782a-rvqsn"]
Nov 03 21:34:01.449461 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:01.449431    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:34:01.449461 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:01.449452    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:34:01.449715 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:01.449539    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:34:01.450426 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:01.450403    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:34:01.457760 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:01.457732    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfw4k\" (UniqueName: \"kubernetes.io/projected/a55e0e3c-bdb8-4ed0-9756-99a0f952584d-kube-api-access-dfw4k\") pod \"maas-api-revoke-sub-1c39782a-rvqsn\" (UID: \"a55e0e3c-bdb8-4ed0-9756-99a0f952584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c39782a-rvqsn"
Nov 03 21:34:01.457889 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:01.457868    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55e0e3c-bdb8-4ed0-9756-99a0f952584d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c39782a-rvqsn\" (UID: \"a55e0e3c-bdb8-4ed0-9756-99a0f952584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c39782a-rvqsn"
Nov 03 21:34:01.457958 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:01.457939    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55e0e3c-bdb8-4ed0-9756-99a0f952584d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c39782a-rvqsn\" (UID: \"a55e0e3c-bdb8-4ed0-9756-99a0f952584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c39782a-rvqsn"
Nov 03 21:34:01.558664 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:01.558636    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55e0e3c-bdb8-4ed0-9756-99a0f952584d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c39782a-rvqsn\" (UID: \"a55e0e3c-bdb8-4ed0-9756-99a0f952584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c39782a-rvqsn"
Nov 03 21:34:01.558826 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:01.558679    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55e0e3c-bdb8-4ed0-9756-99a0f952584d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c39782a-rvqsn\" (UID: \"a55e0e3c-bdb8-4ed0-9756-99a0f952584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c39782a-rvqsn"
Nov 03 21:34:01.558942 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:01.558917    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfw4k\" (UniqueName: \"kubernetes.io/projected/a55e0e3c-bdb8-4ed0-9756-99a0f952584d-kube-api-access-dfw4k\") pod \"maas-api-revoke-sub-1c39782a-rvqsn\" (UID: \"a55e0e3c-bdb8-4ed0-9756-99a0f952584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c39782a-rvqsn"
Nov 03 21:34:01.559360 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:01.559334    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55e0e3c-bdb8-4ed0-9756-99a0f952584d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c39782a-rvqsn\" (UID: \"a55e0e3c-bdb8-4ed0-9756-99a0f952584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c39782a-rvqsn"
Nov 03 21:34:01.561506 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:01.561480    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55e0e3c-bdb8-4ed0-9756-99a0f952584d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c39782a-rvqsn\" (UID: \"a55e0e3c-bdb8-4ed0-9756-99a0f952584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c39782a-rvqsn"
Nov 03 21:34:01.618171 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:01.618084    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfw4k\" (UniqueName: \"kubernetes.io/projected/a55e0e3c-bdb8-4ed0-9756-99a0f952584d-kube-api-access-dfw4k\") pod \"maas-api-revoke-sub-1c39782a-rvqsn\" (UID: \"a55e0e3c-bdb8-4ed0-9756-99a0f952584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c39782a-rvqsn"
Nov 03 21:34:01.662473 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:01.662436    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c39782a-rvqsn"
Nov 03 21:34:02.040682 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:02.040645    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c39782a-rvqsn"]
Nov 03 21:34:02.041071 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:34:02.041047    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda55e0e3c_bdb8_4ed0_9756_99a0f952584d.slice/crio-72d2858ef8e9c404c6fe14409e5bcc6407b100f71ad413c96878586e3ad39dba WatchSource:0}: Error finding container 72d2858ef8e9c404c6fe14409e5bcc6407b100f71ad413c96878586e3ad39dba: Status 404 returned error can't find the container with id 72d2858ef8e9c404c6fe14409e5bcc6407b100f71ad413c96878586e3ad39dba
Nov 03 21:34:02.521325 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:02.521279    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c39782a-rvqsn" event={"ID":"a55e0e3c-bdb8-4ed0-9756-99a0f952584d","Type":"ContainerStarted","Data":"d3eb856724542f530cd59ee0abc012586480ee105043156ab778a42e292ccca6"}
Nov 03 21:34:02.521750 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:02.521358    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c39782a-rvqsn" event={"ID":"a55e0e3c-bdb8-4ed0-9756-99a0f952584d","Type":"ContainerStarted","Data":"72d2858ef8e9c404c6fe14409e5bcc6407b100f71ad413c96878586e3ad39dba"}
Nov 03 21:34:02.634988 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:02.634891    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c39782a-rvqsn" podStartSLOduration=1.63487721 podStartE2EDuration="1.63487721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:02.633151781 +0000 UTC m=+1157.127073667" watchObservedRunningTime="2026-11-03 21:34:02.63487721 +0000 UTC m=+1157.128799085"
Nov 03 21:34:03.530300 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:03.530269    5053 generic.go:356] "Generic (PLEG): container finished" podID="a55e0e3c-bdb8-4ed0-9756-99a0f952584d" containerID="d3eb856724542f530cd59ee0abc012586480ee105043156ab778a42e292ccca6" exitCode=0
Nov 03 21:34:03.530677 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:03.530361    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c39782a-rvqsn" event={"ID":"a55e0e3c-bdb8-4ed0-9756-99a0f952584d","Type":"ContainerDied","Data":"d3eb856724542f530cd59ee0abc012586480ee105043156ab778a42e292ccca6"}
Nov 03 21:34:04.692179 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:04.692145    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c39782a-rvqsn"
Nov 03 21:34:04.797142 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:04.797054    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a55e0e3c-bdb8-4ed0-9756-99a0f952584d-kube-api-access-dfw4k\" (UniqueName: \"kubernetes.io/projected/a55e0e3c-bdb8-4ed0-9756-99a0f952584d-kube-api-access-dfw4k\") pod \"a55e0e3c-bdb8-4ed0-9756-99a0f952584d\" (UID: \"a55e0e3c-bdb8-4ed0-9756-99a0f952584d\") "
Nov 03 21:34:04.797359 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:04.797263    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a55e0e3c-bdb8-4ed0-9756-99a0f952584d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55e0e3c-bdb8-4ed0-9756-99a0f952584d-maas-api-openshift-ca\") pod \"a55e0e3c-bdb8-4ed0-9756-99a0f952584d\" (UID: \"a55e0e3c-bdb8-4ed0-9756-99a0f952584d\") "
Nov 03 21:34:04.797359 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:04.797314    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a55e0e3c-bdb8-4ed0-9756-99a0f952584d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55e0e3c-bdb8-4ed0-9756-99a0f952584d-maas-api-xks-ca\") pod \"a55e0e3c-bdb8-4ed0-9756-99a0f952584d\" (UID: \"a55e0e3c-bdb8-4ed0-9756-99a0f952584d\") "
Nov 03 21:34:04.797587 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:04.797567    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a55e0e3c-bdb8-4ed0-9756-99a0f952584d-maas-api-openshift-ca" pod "a55e0e3c-bdb8-4ed0-9756-99a0f952584d" (UID: "a55e0e3c-bdb8-4ed0-9756-99a0f952584d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:34:04.797755 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:04.797739    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55e0e3c-bdb8-4ed0-9756-99a0f952584d-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:04.799418 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:04.799385    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a55e0e3c-bdb8-4ed0-9756-99a0f952584d-kube-api-access-dfw4k" pod "a55e0e3c-bdb8-4ed0-9756-99a0f952584d" (UID: "a55e0e3c-bdb8-4ed0-9756-99a0f952584d"). InnerVolumeSpecName "kube-api-access-dfw4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:34:04.799537 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:04.799433    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a55e0e3c-bdb8-4ed0-9756-99a0f952584d-maas-api-xks-ca" pod "a55e0e3c-bdb8-4ed0-9756-99a0f952584d" (UID: "a55e0e3c-bdb8-4ed0-9756-99a0f952584d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:34:04.899181 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:04.899151    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55e0e3c-bdb8-4ed0-9756-99a0f952584d-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:04.899181 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:04.899178    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfw4k\" (UniqueName: \"kubernetes.io/projected/a55e0e3c-bdb8-4ed0-9756-99a0f952584d-kube-api-access-dfw4k\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:05.544778 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:05.544746    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c39782a-rvqsn"
Nov 03 21:34:05.544984 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:05.544785    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c39782a-rvqsn" event={"ID":"a55e0e3c-bdb8-4ed0-9756-99a0f952584d","Type":"ContainerDied","Data":"72d2858ef8e9c404c6fe14409e5bcc6407b100f71ad413c96878586e3ad39dba"}
Nov 03 21:34:05.544984 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:05.544838    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72d2858ef8e9c404c6fe14409e5bcc6407b100f71ad413c96878586e3ad39dba"
Nov 03 21:34:07.893796 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:07.893759    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ea73bd-zvfss"]
Nov 03 21:34:07.894381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:07.894367    5053 cpu_manager.go:414] "containerMap: removing container" podUID="a55e0e3c-bdb8-4ed0-9756-99a0f952584d" containerName="revoke-keys"
Nov 03 21:34:07.894440 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:07.894383    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a55e0e3c-bdb8-4ed0-9756-99a0f952584d" containerName="revoke-keys"
Nov 03 21:34:07.894481 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:07.894471    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="a55e0e3c-bdb8-4ed0-9756-99a0f952584d" containerName="revoke-keys"
Nov 03 21:34:07.898162 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:07.898147    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ea73bd-zvfss"
Nov 03 21:34:07.955028 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:07.954997    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ea73bd-zvfss"]
Nov 03 21:34:08.000041 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:08.000002    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:34:08.000041 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:08.000039    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:34:08.000333 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:08.000161    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:34:08.002060 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:08.002030    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:34:08.038307 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:08.038267    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j42c8\" (UniqueName: \"kubernetes.io/projected/e1809882-daeb-48eb-82ac-76797b259c1c-kube-api-access-j42c8\") pod \"maas-api-revoke-sub-73ea73bd-zvfss\" (UID: \"e1809882-daeb-48eb-82ac-76797b259c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ea73bd-zvfss"
Nov 03 21:34:08.038479 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:08.038439    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1809882-daeb-48eb-82ac-76797b259c1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73ea73bd-zvfss\" (UID: \"e1809882-daeb-48eb-82ac-76797b259c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ea73bd-zvfss"
Nov 03 21:34:08.038479 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:08.038460    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1809882-daeb-48eb-82ac-76797b259c1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73ea73bd-zvfss\" (UID: \"e1809882-daeb-48eb-82ac-76797b259c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ea73bd-zvfss"
Nov 03 21:34:08.139133 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:08.139092    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1809882-daeb-48eb-82ac-76797b259c1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73ea73bd-zvfss\" (UID: \"e1809882-daeb-48eb-82ac-76797b259c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ea73bd-zvfss"
Nov 03 21:34:08.139133 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:08.139134    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1809882-daeb-48eb-82ac-76797b259c1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73ea73bd-zvfss\" (UID: \"e1809882-daeb-48eb-82ac-76797b259c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ea73bd-zvfss"
Nov 03 21:34:08.139421 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:08.139315    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j42c8\" (UniqueName: \"kubernetes.io/projected/e1809882-daeb-48eb-82ac-76797b259c1c-kube-api-access-j42c8\") pod \"maas-api-revoke-sub-73ea73bd-zvfss\" (UID: \"e1809882-daeb-48eb-82ac-76797b259c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ea73bd-zvfss"
Nov 03 21:34:08.139816 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:08.139782    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1809882-daeb-48eb-82ac-76797b259c1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73ea73bd-zvfss\" (UID: \"e1809882-daeb-48eb-82ac-76797b259c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ea73bd-zvfss"
Nov 03 21:34:08.141807 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:08.141785    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1809882-daeb-48eb-82ac-76797b259c1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73ea73bd-zvfss\" (UID: \"e1809882-daeb-48eb-82ac-76797b259c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ea73bd-zvfss"
Nov 03 21:34:08.195960 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:08.195928    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j42c8\" (UniqueName: \"kubernetes.io/projected/e1809882-daeb-48eb-82ac-76797b259c1c-kube-api-access-j42c8\") pod \"maas-api-revoke-sub-73ea73bd-zvfss\" (UID: \"e1809882-daeb-48eb-82ac-76797b259c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ea73bd-zvfss"
Nov 03 21:34:08.212321 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:08.212291    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ea73bd-zvfss"
Nov 03 21:34:08.595681 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:08.595641    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ea73bd-zvfss"]
Nov 03 21:34:09.574515 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:09.574481    5053 generic.go:356] "Generic (PLEG): container finished" podID="e1809882-daeb-48eb-82ac-76797b259c1c" containerID="5fefee905ea404b84bc89925f77cdda5eeecd5bea188bb94592d1a85956c407f" exitCode=0
Nov 03 21:34:09.574879 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:09.574562    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ea73bd-zvfss" event={"ID":"e1809882-daeb-48eb-82ac-76797b259c1c","Type":"ContainerDied","Data":"5fefee905ea404b84bc89925f77cdda5eeecd5bea188bb94592d1a85956c407f"}
Nov 03 21:34:09.574879 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:09.574611    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ea73bd-zvfss" event={"ID":"e1809882-daeb-48eb-82ac-76797b259c1c","Type":"ContainerStarted","Data":"7323ad5b9b68c194535ebd0583c203bfd5980e163fe0fa965e512b669c175f05"}
Nov 03 21:34:10.749035 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:10.749009    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ea73bd-zvfss"
Nov 03 21:34:10.871559 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:10.871478    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1809882-daeb-48eb-82ac-76797b259c1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1809882-daeb-48eb-82ac-76797b259c1c-maas-api-openshift-ca\") pod \"e1809882-daeb-48eb-82ac-76797b259c1c\" (UID: \"e1809882-daeb-48eb-82ac-76797b259c1c\") "
Nov 03 21:34:10.871559 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:10.871547    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1809882-daeb-48eb-82ac-76797b259c1c-kube-api-access-j42c8\" (UniqueName: \"kubernetes.io/projected/e1809882-daeb-48eb-82ac-76797b259c1c-kube-api-access-j42c8\") pod \"e1809882-daeb-48eb-82ac-76797b259c1c\" (UID: \"e1809882-daeb-48eb-82ac-76797b259c1c\") "
Nov 03 21:34:10.871750 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:10.871657    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1809882-daeb-48eb-82ac-76797b259c1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1809882-daeb-48eb-82ac-76797b259c1c-maas-api-xks-ca\") pod \"e1809882-daeb-48eb-82ac-76797b259c1c\" (UID: \"e1809882-daeb-48eb-82ac-76797b259c1c\") "
Nov 03 21:34:10.871985 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:10.871958    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1809882-daeb-48eb-82ac-76797b259c1c-maas-api-openshift-ca" pod "e1809882-daeb-48eb-82ac-76797b259c1c" (UID: "e1809882-daeb-48eb-82ac-76797b259c1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:34:10.872204 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:10.872187    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1809882-daeb-48eb-82ac-76797b259c1c-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:10.874049 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:10.874024    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1809882-daeb-48eb-82ac-76797b259c1c-maas-api-xks-ca" pod "e1809882-daeb-48eb-82ac-76797b259c1c" (UID: "e1809882-daeb-48eb-82ac-76797b259c1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:34:10.874373 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:10.874347    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1809882-daeb-48eb-82ac-76797b259c1c-kube-api-access-j42c8" pod "e1809882-daeb-48eb-82ac-76797b259c1c" (UID: "e1809882-daeb-48eb-82ac-76797b259c1c"). InnerVolumeSpecName "kube-api-access-j42c8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:34:10.938676 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:34:10.938639    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:34:10.973931 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:10.973901    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j42c8\" (UniqueName: \"kubernetes.io/projected/e1809882-daeb-48eb-82ac-76797b259c1c-kube-api-access-j42c8\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:10.973931 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:10.973931    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1809882-daeb-48eb-82ac-76797b259c1c-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:11.589216 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:11.589179    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ea73bd-zvfss"
Nov 03 21:34:11.589427 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:11.589217    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ea73bd-zvfss" event={"ID":"e1809882-daeb-48eb-82ac-76797b259c1c","Type":"ContainerDied","Data":"7323ad5b9b68c194535ebd0583c203bfd5980e163fe0fa965e512b669c175f05"}
Nov 03 21:34:11.589427 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:11.589285    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7323ad5b9b68c194535ebd0583c203bfd5980e163fe0fa965e512b669c175f05"
Nov 03 21:34:14.385263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.351991    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"]
Nov 03 21:34:14.386880 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.386847    5053 cpu_manager.go:414] "containerMap: removing container" podUID="e1809882-daeb-48eb-82ac-76797b259c1c" containerName="revoke-keys"
Nov 03 21:34:14.387030 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.387016    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1809882-daeb-48eb-82ac-76797b259c1c" containerName="revoke-keys"
Nov 03 21:34:14.387336 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.387319    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1809882-daeb-48eb-82ac-76797b259c1c" containerName="revoke-keys"
Nov 03 21:34:14.433482 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.428656    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"]
Nov 03 21:34:14.454499 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.454461    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.500520 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.500490    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"]
Nov 03 21:34:14.500897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.500879    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"]
Nov 03 21:34:14.501143 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.501129    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.561531 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.560686    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.561531 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.560756    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720802a7-8dda-4dcc-8deb-d764518f49c1-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.561531 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.560799    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.561531 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.560837    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.561531 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.560863    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.561531 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.560922    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.561531 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.560978    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.561531 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.561019    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pdvt\" (UniqueName: \"kubernetes.io/projected/c9cd9ef8-5f64-442e-b66a-f6e399137667-kube-api-access-4pdvt\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.561531 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.561066    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9cd9ef8-5f64-442e-b66a-f6e399137667-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.561531 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.561119    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.561531 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.561145    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72vrj\" (UniqueName: \"kubernetes.io/projected/720802a7-8dda-4dcc-8deb-d764518f49c1-kube-api-access-72vrj\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.561531 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.561167    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.561531 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.561194    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.561531 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.561273    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.561531 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.561309    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.561531 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.561350    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.561531 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.561379    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.561531 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.561412    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.700264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.695590    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.700264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.695679    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720802a7-8dda-4dcc-8deb-d764518f49c1-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.700264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.695736    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.700264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.695771    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.700264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.695815    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.700264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.695889    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.700264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.695960    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.700264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.696006    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pdvt\" (UniqueName: \"kubernetes.io/projected/c9cd9ef8-5f64-442e-b66a-f6e399137667-kube-api-access-4pdvt\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.700264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.696066    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9cd9ef8-5f64-442e-b66a-f6e399137667-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.700264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.696133    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.700264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.696160    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72vrj\" (UniqueName: \"kubernetes.io/projected/720802a7-8dda-4dcc-8deb-d764518f49c1-kube-api-access-72vrj\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.700264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.696197    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.700264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.696229    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.700264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.696309    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.700264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.696358    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.700264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.696416    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.700264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.696443    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.700264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.696477    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.700264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.697052    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.700264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.697495    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.700264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.698327    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720802a7-8dda-4dcc-8deb-d764518f49c1-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.700264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.698614    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.714688 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.705534    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.714688 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.713142    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9cd9ef8-5f64-442e-b66a-f6e399137667-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.714688 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.713411    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.741262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.715802    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.741262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.716280    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.751262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.746560    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.751262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.746834    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.778913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.778876    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.783946 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.783912    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72vrj\" (UniqueName: \"kubernetes.io/projected/720802a7-8dda-4dcc-8deb-d764518f49c1-kube-api-access-72vrj\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.809375 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.793836    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.810783 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.810754    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.814805 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.814755    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.837260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.832401    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pdvt\" (UniqueName: \"kubernetes.io/projected/c9cd9ef8-5f64-442e-b66a-f6e399137667-kube-api-access-4pdvt\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:14.837796 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.837771    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:14.873262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:14.872473    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:15.135834 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:15.135777    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:15.346298 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:15.346268    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"]
Nov 03 21:34:15.347114 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:34:15.347073    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9cd9ef8_5f64_442e_b66a_f6e399137667.slice/crio-e4f3277e61bf4186939fd507e58c393e249e5bd32f24f61dfb1bc3c087cab092 WatchSource:0}: Error finding container e4f3277e61bf4186939fd507e58c393e249e5bd32f24f61dfb1bc3c087cab092: Status 404 returned error can't find the container with id e4f3277e61bf4186939fd507e58c393e249e5bd32f24f61dfb1bc3c087cab092
Nov 03 21:34:15.350009 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:15.349970    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:34:15.350095 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:15.350055    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:34:15.350133 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:15.350096    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:34:15.527866 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:15.527832    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"]
Nov 03 21:34:15.528394 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:34:15.528369    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720802a7_8dda_4dcc_8deb_d764518f49c1.slice/crio-970efc0b78f32654cd1c2a939f62fec18b9f258eb7f1a723c3ca16057ae97d60 WatchSource:0}: Error finding container 970efc0b78f32654cd1c2a939f62fec18b9f258eb7f1a723c3ca16057ae97d60: Status 404 returned error can't find the container with id 970efc0b78f32654cd1c2a939f62fec18b9f258eb7f1a723c3ca16057ae97d60
Nov 03 21:34:15.531896 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:15.531851    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:34:15.531995 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:15.531940    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:34:15.531995 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:15.531978    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:34:15.689472 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:15.689430    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k" event={"ID":"720802a7-8dda-4dcc-8deb-d764518f49c1","Type":"ContainerStarted","Data":"a2dbc742beb4de19a46484ae484f3f33d51c4160142da69d83490929a3f21094"}
Nov 03 21:34:15.689636 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:15.689494    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k" event={"ID":"720802a7-8dda-4dcc-8deb-d764518f49c1","Type":"ContainerStarted","Data":"970efc0b78f32654cd1c2a939f62fec18b9f258eb7f1a723c3ca16057ae97d60"}
Nov 03 21:34:15.691481 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:15.691450    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh" event={"ID":"c9cd9ef8-5f64-442e-b66a-f6e399137667","Type":"ContainerStarted","Data":"af154710f0e2effa9373102b0cf0944216cb6509790064e21d20ccabb338fe28"}
Nov 03 21:34:15.691589 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:15.691508    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh" event={"ID":"c9cd9ef8-5f64-442e-b66a-f6e399137667","Type":"ContainerStarted","Data":"e4f3277e61bf4186939fd507e58c393e249e5bd32f24f61dfb1bc3c087cab092"}
Nov 03 21:34:15.806972 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:15.806916    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k" podStartSLOduration=2.806901624 podStartE2EDuration="2.806901624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:15.803904298 +0000 UTC m=+1170.297826181" watchObservedRunningTime="2026-11-03 21:34:15.806901624 +0000 UTC m=+1170.300823498"
Nov 03 21:34:15.873605 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:15.873516    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:15.922621 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:15.922563    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh" podStartSLOduration=2.922544224 podStartE2EDuration="2.922544224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:15.920817559 +0000 UTC m=+1170.414739431" watchObservedRunningTime="2026-11-03 21:34:15.922544224 +0000 UTC m=+1170.416466105"
Nov 03 21:34:16.136841 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:16.136736    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:16.137957 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:16.137929    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 21:34:16.138134 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:16.137989    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k" podUID="720802a7-8dda-4dcc-8deb-d764518f49c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 21:34:16.874245 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:16.874197    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 21:34:16.874640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:16.874267    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh" podUID="c9cd9ef8-5f64-442e-b66a-f6e399137667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 21:34:17.148325 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:17.141747    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 21:34:17.148325 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:17.141814    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k" podUID="720802a7-8dda-4dcc-8deb-d764518f49c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 21:34:17.747990 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:17.747937    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e52642e-l6wlq"]
Nov 03 21:34:17.769931 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:17.764982    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e52642e-l6wlq"
Nov 03 21:34:17.784443 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:17.784417    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e52642e-l6wlq"]
Nov 03 21:34:17.822271 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:17.822214    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:34:17.842673 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:17.842636    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:34:17.843684 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:17.843089    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:34:17.843819 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:17.843127    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:34:17.873701 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:17.873672    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 21:34:17.873956 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:17.873928    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh" podUID="c9cd9ef8-5f64-442e-b66a-f6e399137667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 21:34:17.879707 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:17.879680    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crvdb\" (UniqueName: \"kubernetes.io/projected/c4f71ae5-f966-4a7b-9181-6151aa57f43f-kube-api-access-crvdb\") pod \"maas-api-revoke-sub-8e52642e-l6wlq\" (UID: \"c4f71ae5-f966-4a7b-9181-6151aa57f43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e52642e-l6wlq"
Nov 03 21:34:17.880698 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:17.880676    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4f71ae5-f966-4a7b-9181-6151aa57f43f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e52642e-l6wlq\" (UID: \"c4f71ae5-f966-4a7b-9181-6151aa57f43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e52642e-l6wlq"
Nov 03 21:34:17.880932 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:17.880915    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4f71ae5-f966-4a7b-9181-6151aa57f43f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e52642e-l6wlq\" (UID: \"c4f71ae5-f966-4a7b-9181-6151aa57f43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e52642e-l6wlq"
Nov 03 21:34:17.982270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:17.982201    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4f71ae5-f966-4a7b-9181-6151aa57f43f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e52642e-l6wlq\" (UID: \"c4f71ae5-f966-4a7b-9181-6151aa57f43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e52642e-l6wlq"
Nov 03 21:34:17.982451 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:17.982317    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crvdb\" (UniqueName: \"kubernetes.io/projected/c4f71ae5-f966-4a7b-9181-6151aa57f43f-kube-api-access-crvdb\") pod \"maas-api-revoke-sub-8e52642e-l6wlq\" (UID: \"c4f71ae5-f966-4a7b-9181-6151aa57f43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e52642e-l6wlq"
Nov 03 21:34:17.982534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:17.982490    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4f71ae5-f966-4a7b-9181-6151aa57f43f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e52642e-l6wlq\" (UID: \"c4f71ae5-f966-4a7b-9181-6151aa57f43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e52642e-l6wlq"
Nov 03 21:34:17.983350 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:17.983320    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4f71ae5-f966-4a7b-9181-6151aa57f43f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e52642e-l6wlq\" (UID: \"c4f71ae5-f966-4a7b-9181-6151aa57f43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e52642e-l6wlq"
Nov 03 21:34:17.984783 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:17.984762    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4f71ae5-f966-4a7b-9181-6151aa57f43f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e52642e-l6wlq\" (UID: \"c4f71ae5-f966-4a7b-9181-6151aa57f43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e52642e-l6wlq"
Nov 03 21:34:18.069643 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:18.069534    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crvdb\" (UniqueName: \"kubernetes.io/projected/c4f71ae5-f966-4a7b-9181-6151aa57f43f-kube-api-access-crvdb\") pod \"maas-api-revoke-sub-8e52642e-l6wlq\" (UID: \"c4f71ae5-f966-4a7b-9181-6151aa57f43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e52642e-l6wlq"
Nov 03 21:34:18.086321 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:18.086293    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e52642e-l6wlq"
Nov 03 21:34:18.148263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:18.147712    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 21:34:18.148263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:18.147778    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k" podUID="720802a7-8dda-4dcc-8deb-d764518f49c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 21:34:18.581014 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:18.580984    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e52642e-l6wlq"]
Nov 03 21:34:18.581396 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:34:18.581368    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f71ae5_f966_4a7b_9181_6151aa57f43f.slice/crio-f5d95761a38bc711be9ceb93cffd6a11041738e3ad9b1c71796a92bf713fe99f WatchSource:0}: Error finding container f5d95761a38bc711be9ceb93cffd6a11041738e3ad9b1c71796a92bf713fe99f: Status 404 returned error can't find the container with id f5d95761a38bc711be9ceb93cffd6a11041738e3ad9b1c71796a92bf713fe99f
Nov 03 21:34:18.762923 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:18.762770    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e52642e-l6wlq" event={"ID":"c4f71ae5-f966-4a7b-9181-6151aa57f43f","Type":"ContainerStarted","Data":"f5d95761a38bc711be9ceb93cffd6a11041738e3ad9b1c71796a92bf713fe99f"}
Nov 03 21:34:18.873332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:18.873114    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 21:34:18.873332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:18.873159    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh" podUID="c9cd9ef8-5f64-442e-b66a-f6e399137667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 21:34:18.960263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:18.959169    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"]
Nov 03 21:34:18.968095 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:18.968064    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:18.995368 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:18.995339    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:18.995539 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:18.995420    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:18.995539 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:18.995472    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:18.995539 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:18.995516    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:18.995705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:18.995650    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:18.995749 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:18.995717    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:18.995785 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:18.995759    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:18.995820 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:18.995783    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wwtl\" (UniqueName: \"kubernetes.io/projected/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-kube-api-access-5wwtl\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:18.995820 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:18.995813    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:19.061263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.054819    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"]
Nov 03 21:34:19.096911 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.096869    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:19.096911 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.096917    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:19.097154 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.096934    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wwtl\" (UniqueName: \"kubernetes.io/projected/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-kube-api-access-5wwtl\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:19.097154 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.096965    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:19.097154 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.097061    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:19.097154 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.097076    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:19.097154 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.097106    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:19.097154 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.097135    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:19.097465 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.097187    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:19.097865 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.097837    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:19.098039 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.098018    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:19.098107 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.098086    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:19.098183 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.098165    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:19.098307 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.098288    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:19.100041 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.100017    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:19.127323 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.100502    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:19.168263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.168091    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 21:34:19.168263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.168151    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k" podUID="720802a7-8dda-4dcc-8deb-d764518f49c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 21:34:19.171779 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.171750    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:19.174602 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.174574    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wwtl\" (UniqueName: \"kubernetes.io/projected/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-kube-api-access-5wwtl\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:19.285206 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.285168    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:19.774226 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.774189    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e52642e-l6wlq" event={"ID":"c4f71ae5-f966-4a7b-9181-6151aa57f43f","Type":"ContainerStarted","Data":"8dff7203d1ffca7c345584865a05c3bcb613fa050a78b315852338f160a8b22d"}
Nov 03 21:34:19.777914 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.777889    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"]
Nov 03 21:34:19.778635 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:34:19.778608    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a2dddd_5328_44b5_a5ae_b82bfcd8d9aa.slice/crio-c34e4c700c934b769f0412c09828262803e9c66486e16a21f8c8e7966f17c24a WatchSource:0}: Error finding container c34e4c700c934b769f0412c09828262803e9c66486e16a21f8c8e7966f17c24a: Status 404 returned error can't find the container with id c34e4c700c934b769f0412c09828262803e9c66486e16a21f8c8e7966f17c24a
Nov 03 21:34:19.780907 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.780875    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:34:19.780952 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.780940    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:34:19.780990 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.780966    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:34:19.878875 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.878829    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 21:34:19.880059 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.880018    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh" podUID="c9cd9ef8-5f64-442e-b66a-f6e399137667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 21:34:19.919282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:19.915047    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e52642e-l6wlq" podStartSLOduration=2.915023231 podStartE2EDuration="2.915023231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:19.892796502 +0000 UTC m=+1174.386718379" watchObservedRunningTime="2026-11-03 21:34:19.915023231 +0000 UTC m=+1174.408945107"
Nov 03 21:34:20.142927 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:20.136672    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 21:34:20.142927 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:20.136735    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k" podUID="720802a7-8dda-4dcc-8deb-d764518f49c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 21:34:20.793681 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:20.793587    5053 generic.go:356] "Generic (PLEG): container finished" podID="c4f71ae5-f966-4a7b-9181-6151aa57f43f" containerID="8dff7203d1ffca7c345584865a05c3bcb613fa050a78b315852338f160a8b22d" exitCode=0
Nov 03 21:34:20.793874 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:20.793700    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e52642e-l6wlq" event={"ID":"c4f71ae5-f966-4a7b-9181-6151aa57f43f","Type":"ContainerDied","Data":"8dff7203d1ffca7c345584865a05c3bcb613fa050a78b315852338f160a8b22d"}
Nov 03 21:34:20.795807 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:20.795780    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z" event={"ID":"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa","Type":"ContainerStarted","Data":"c9ae10655d87ddcaea7236fc2fb980e11c5677bfb8a3addcc57f4b6edafe51b8"}
Nov 03 21:34:20.795941 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:20.795829    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z" event={"ID":"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa","Type":"ContainerStarted","Data":"c34e4c700c934b769f0412c09828262803e9c66486e16a21f8c8e7966f17c24a"}
Nov 03 21:34:20.878261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:20.876647    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 21:34:20.878261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:20.876726    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh" podUID="c9cd9ef8-5f64-442e-b66a-f6e399137667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 21:34:21.085786 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:21.085631    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z" podStartSLOduration=3.085608492 podStartE2EDuration="3.085608492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:21.082635986 +0000 UTC m=+1175.576557873" watchObservedRunningTime="2026-11-03 21:34:21.085608492 +0000 UTC m=+1175.579530369"
Nov 03 21:34:21.137749 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:21.137712    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 21:34:21.138040 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:21.137999    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k" podUID="720802a7-8dda-4dcc-8deb-d764518f49c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 21:34:21.286345 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:21.286317    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:21.287399 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:21.287374    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 21:34:21.287542 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:21.287425    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z" podUID="b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 21:34:21.876222 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:21.875648    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 21:34:21.876222 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:21.875710    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh" podUID="c9cd9ef8-5f64-442e-b66a-f6e399137667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 21:34:22.164570 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:22.164511    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 21:34:22.165217 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:22.164814    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k" podUID="720802a7-8dda-4dcc-8deb-d764518f49c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 21:34:22.169105 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:22.169083    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e52642e-l6wlq"
Nov 03 21:34:22.253692 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:22.253651    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4f71ae5-f966-4a7b-9181-6151aa57f43f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4f71ae5-f966-4a7b-9181-6151aa57f43f-maas-api-xks-ca\") pod \"c4f71ae5-f966-4a7b-9181-6151aa57f43f\" (UID: \"c4f71ae5-f966-4a7b-9181-6151aa57f43f\") "
Nov 03 21:34:22.253898 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:22.253750    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4f71ae5-f966-4a7b-9181-6151aa57f43f-kube-api-access-crvdb\" (UniqueName: \"kubernetes.io/projected/c4f71ae5-f966-4a7b-9181-6151aa57f43f-kube-api-access-crvdb\") pod \"c4f71ae5-f966-4a7b-9181-6151aa57f43f\" (UID: \"c4f71ae5-f966-4a7b-9181-6151aa57f43f\") "
Nov 03 21:34:22.253973 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:22.253958    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4f71ae5-f966-4a7b-9181-6151aa57f43f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4f71ae5-f966-4a7b-9181-6151aa57f43f-maas-api-openshift-ca\") pod \"c4f71ae5-f966-4a7b-9181-6151aa57f43f\" (UID: \"c4f71ae5-f966-4a7b-9181-6151aa57f43f\") "
Nov 03 21:34:22.254485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:22.254450    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f71ae5-f966-4a7b-9181-6151aa57f43f-maas-api-openshift-ca" pod "c4f71ae5-f966-4a7b-9181-6151aa57f43f" (UID: "c4f71ae5-f966-4a7b-9181-6151aa57f43f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:34:22.254598 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:22.254563    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4f71ae5-f966-4a7b-9181-6151aa57f43f-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:22.256069 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:22.256045    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4f71ae5-f966-4a7b-9181-6151aa57f43f-maas-api-xks-ca" pod "c4f71ae5-f966-4a7b-9181-6151aa57f43f" (UID: "c4f71ae5-f966-4a7b-9181-6151aa57f43f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:34:22.260741 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:22.260713    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f71ae5-f966-4a7b-9181-6151aa57f43f-kube-api-access-crvdb" pod "c4f71ae5-f966-4a7b-9181-6151aa57f43f" (UID: "c4f71ae5-f966-4a7b-9181-6151aa57f43f"). InnerVolumeSpecName "kube-api-access-crvdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:34:22.287644 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:22.287550    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 21:34:22.287644 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:22.287604    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z" podUID="b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 21:34:22.356385 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:22.356300    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crvdb\" (UniqueName: \"kubernetes.io/projected/c4f71ae5-f966-4a7b-9181-6151aa57f43f-kube-api-access-crvdb\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:22.356385 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:22.356330    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4f71ae5-f966-4a7b-9181-6151aa57f43f-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:22.839637 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:22.839608    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e52642e-l6wlq"
Nov 03 21:34:22.839637 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:22.839629    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e52642e-l6wlq" event={"ID":"c4f71ae5-f966-4a7b-9181-6151aa57f43f","Type":"ContainerDied","Data":"f5d95761a38bc711be9ceb93cffd6a11041738e3ad9b1c71796a92bf713fe99f"}
Nov 03 21:34:22.839897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:22.839699    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5d95761a38bc711be9ceb93cffd6a11041738e3ad9b1c71796a92bf713fe99f"
Nov 03 21:34:22.877376 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:22.877348    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:22.877649 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:22.877627    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:22.878302 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:22.878285    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:34:23.141088 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:23.141010    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:23.141505 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:23.141482    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:23.142146 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:23.142132    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:34:23.285875 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:23.285843    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 21:34:23.286046 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:23.285904    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z" podUID="b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 21:34:24.286635 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:24.286598    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 21:34:24.287121 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:24.286697    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z" podUID="b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 21:34:25.286310 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:25.286278    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 21:34:25.286493 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:25.286334    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z" podUID="b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 21:34:25.632935 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:25.632848    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb40886b-64qgl"]
Nov 03 21:34:25.633839 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:25.633821    5053 cpu_manager.go:414] "containerMap: removing container" podUID="c4f71ae5-f966-4a7b-9181-6151aa57f43f" containerName="revoke-keys"
Nov 03 21:34:25.633893 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:25.633842    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4f71ae5-f966-4a7b-9181-6151aa57f43f" containerName="revoke-keys"
Nov 03 21:34:25.633985 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:25.633975    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4f71ae5-f966-4a7b-9181-6151aa57f43f" containerName="revoke-keys"
Nov 03 21:34:25.638377 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:25.638359    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb40886b-64qgl"
Nov 03 21:34:25.669529 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:25.669497    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb40886b-64qgl"]
Nov 03 21:34:25.690022 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:25.689995    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:34:25.696753 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:25.696720    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08e9108a-8670-47b9-a15b-a530895231e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb40886b-64qgl\" (UID: \"08e9108a-8670-47b9-a15b-a530895231e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb40886b-64qgl"
Nov 03 21:34:25.696890 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:25.696777    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64shh\" (UniqueName: \"kubernetes.io/projected/08e9108a-8670-47b9-a15b-a530895231e1-kube-api-access-64shh\") pod \"maas-api-revoke-sub-bb40886b-64qgl\" (UID: \"08e9108a-8670-47b9-a15b-a530895231e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb40886b-64qgl"
Nov 03 21:34:25.696930 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:25.696879    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08e9108a-8670-47b9-a15b-a530895231e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb40886b-64qgl\" (UID: \"08e9108a-8670-47b9-a15b-a530895231e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb40886b-64qgl"
Nov 03 21:34:25.724263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:25.723454    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:34:25.724263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:25.723515    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:34:25.724263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:25.723618    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:34:25.797805 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:25.797766    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08e9108a-8670-47b9-a15b-a530895231e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb40886b-64qgl\" (UID: \"08e9108a-8670-47b9-a15b-a530895231e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb40886b-64qgl"
Nov 03 21:34:25.798015 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:25.797827    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64shh\" (UniqueName: \"kubernetes.io/projected/08e9108a-8670-47b9-a15b-a530895231e1-kube-api-access-64shh\") pod \"maas-api-revoke-sub-bb40886b-64qgl\" (UID: \"08e9108a-8670-47b9-a15b-a530895231e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb40886b-64qgl"
Nov 03 21:34:25.798015 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:25.797871    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08e9108a-8670-47b9-a15b-a530895231e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb40886b-64qgl\" (UID: \"08e9108a-8670-47b9-a15b-a530895231e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb40886b-64qgl"
Nov 03 21:34:25.798448 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:25.798422    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08e9108a-8670-47b9-a15b-a530895231e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb40886b-64qgl\" (UID: \"08e9108a-8670-47b9-a15b-a530895231e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb40886b-64qgl"
Nov 03 21:34:25.800375 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:25.800353    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08e9108a-8670-47b9-a15b-a530895231e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb40886b-64qgl\" (UID: \"08e9108a-8670-47b9-a15b-a530895231e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb40886b-64qgl"
Nov 03 21:34:25.853648 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:25.853612    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64shh\" (UniqueName: \"kubernetes.io/projected/08e9108a-8670-47b9-a15b-a530895231e1-kube-api-access-64shh\") pod \"maas-api-revoke-sub-bb40886b-64qgl\" (UID: \"08e9108a-8670-47b9-a15b-a530895231e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb40886b-64qgl"
Nov 03 21:34:25.953229 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:25.953192    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb40886b-64qgl"
Nov 03 21:34:26.286006 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:26.285924    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 21:34:26.286006 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:26.285993    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z" podUID="b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 21:34:26.333629 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:26.333604    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb40886b-64qgl"]
Nov 03 21:34:26.334944 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:34:26.334914    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e9108a_8670_47b9_a15b_a530895231e1.slice/crio-5215662b760d7228246d1e78ab62a1ecbda9d5ad2a46cbc394b21b3b02ca7869 WatchSource:0}: Error finding container 5215662b760d7228246d1e78ab62a1ecbda9d5ad2a46cbc394b21b3b02ca7869: Status 404 returned error can't find the container with id 5215662b760d7228246d1e78ab62a1ecbda9d5ad2a46cbc394b21b3b02ca7869
Nov 03 21:34:26.873815 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:26.873704    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb40886b-64qgl" event={"ID":"08e9108a-8670-47b9-a15b-a530895231e1","Type":"ContainerStarted","Data":"1230465ecd422e6297a2ce3104b0ada508aa8155c8c658fb5c5fabe2b4a13c9a"}
Nov 03 21:34:26.873815 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:26.873771    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb40886b-64qgl" event={"ID":"08e9108a-8670-47b9-a15b-a530895231e1","Type":"ContainerStarted","Data":"5215662b760d7228246d1e78ab62a1ecbda9d5ad2a46cbc394b21b3b02ca7869"}
Nov 03 21:34:26.989736 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:26.989679    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb40886b-64qgl" podStartSLOduration=1.989666228 podStartE2EDuration="1.989666228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:26.985633738 +0000 UTC m=+1181.479555626" watchObservedRunningTime="2026-11-03 21:34:26.989666228 +0000 UTC m=+1181.483588102"
Nov 03 21:34:27.347960 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:27.347484    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:27.348546 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:27.348354    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:27.349727 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:27.349707    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:34:27.882748 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:27.882706    5053 generic.go:356] "Generic (PLEG): container finished" podID="08e9108a-8670-47b9-a15b-a530895231e1" containerID="1230465ecd422e6297a2ce3104b0ada508aa8155c8c658fb5c5fabe2b4a13c9a" exitCode=0
Nov 03 21:34:27.883129 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:27.882739    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb40886b-64qgl" event={"ID":"08e9108a-8670-47b9-a15b-a530895231e1","Type":"ContainerDied","Data":"1230465ecd422e6297a2ce3104b0ada508aa8155c8c658fb5c5fabe2b4a13c9a"}
Nov 03 21:34:28.511436 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:28.511396    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a3ef192-79tzf"]
Nov 03 21:34:28.516327 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:28.516290    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ef192-79tzf"
Nov 03 21:34:28.570399 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:28.570369    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a3ef192-79tzf"]
Nov 03 21:34:28.640259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:28.640199    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmq52\" (UniqueName: \"kubernetes.io/projected/243d8a52-c0be-4581-ab64-bf3897f88f1d-kube-api-access-xmq52\") pod \"maas-api-revoke-sub-1a3ef192-79tzf\" (UID: \"243d8a52-c0be-4581-ab64-bf3897f88f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ef192-79tzf"
Nov 03 21:34:28.640462 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:28.640332    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/243d8a52-c0be-4581-ab64-bf3897f88f1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a3ef192-79tzf\" (UID: \"243d8a52-c0be-4581-ab64-bf3897f88f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ef192-79tzf"
Nov 03 21:34:28.640462 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:28.640370    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/243d8a52-c0be-4581-ab64-bf3897f88f1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a3ef192-79tzf\" (UID: \"243d8a52-c0be-4581-ab64-bf3897f88f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ef192-79tzf"
Nov 03 21:34:28.741495 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:28.741456    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmq52\" (UniqueName: \"kubernetes.io/projected/243d8a52-c0be-4581-ab64-bf3897f88f1d-kube-api-access-xmq52\") pod \"maas-api-revoke-sub-1a3ef192-79tzf\" (UID: \"243d8a52-c0be-4581-ab64-bf3897f88f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ef192-79tzf"
Nov 03 21:34:28.741762 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:28.741732    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/243d8a52-c0be-4581-ab64-bf3897f88f1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a3ef192-79tzf\" (UID: \"243d8a52-c0be-4581-ab64-bf3897f88f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ef192-79tzf"
Nov 03 21:34:28.741849 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:28.741795    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/243d8a52-c0be-4581-ab64-bf3897f88f1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a3ef192-79tzf\" (UID: \"243d8a52-c0be-4581-ab64-bf3897f88f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ef192-79tzf"
Nov 03 21:34:28.742517 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:28.742488    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/243d8a52-c0be-4581-ab64-bf3897f88f1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a3ef192-79tzf\" (UID: \"243d8a52-c0be-4581-ab64-bf3897f88f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ef192-79tzf"
Nov 03 21:34:28.744994 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:28.744969    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/243d8a52-c0be-4581-ab64-bf3897f88f1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a3ef192-79tzf\" (UID: \"243d8a52-c0be-4581-ab64-bf3897f88f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ef192-79tzf"
Nov 03 21:34:28.798287 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:28.798181    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmq52\" (UniqueName: \"kubernetes.io/projected/243d8a52-c0be-4581-ab64-bf3897f88f1d-kube-api-access-xmq52\") pod \"maas-api-revoke-sub-1a3ef192-79tzf\" (UID: \"243d8a52-c0be-4581-ab64-bf3897f88f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ef192-79tzf"
Nov 03 21:34:28.834178 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:28.834144    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ef192-79tzf"
Nov 03 21:34:29.097032 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:29.097005    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb40886b-64qgl"
Nov 03 21:34:29.147894 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:29.147861    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08e9108a-8670-47b9-a15b-a530895231e1-kube-api-access-64shh\" (UniqueName: \"kubernetes.io/projected/08e9108a-8670-47b9-a15b-a530895231e1-kube-api-access-64shh\") pod \"08e9108a-8670-47b9-a15b-a530895231e1\" (UID: \"08e9108a-8670-47b9-a15b-a530895231e1\") "
Nov 03 21:34:29.148100 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:29.147969    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08e9108a-8670-47b9-a15b-a530895231e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08e9108a-8670-47b9-a15b-a530895231e1-maas-api-xks-ca\") pod \"08e9108a-8670-47b9-a15b-a530895231e1\" (UID: \"08e9108a-8670-47b9-a15b-a530895231e1\") "
Nov 03 21:34:29.148194 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:29.148163    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08e9108a-8670-47b9-a15b-a530895231e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08e9108a-8670-47b9-a15b-a530895231e1-maas-api-openshift-ca\") pod \"08e9108a-8670-47b9-a15b-a530895231e1\" (UID: \"08e9108a-8670-47b9-a15b-a530895231e1\") "
Nov 03 21:34:29.148917 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:29.148783    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08e9108a-8670-47b9-a15b-a530895231e1-maas-api-openshift-ca" pod "08e9108a-8670-47b9-a15b-a530895231e1" (UID: "08e9108a-8670-47b9-a15b-a530895231e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:34:29.150629 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:29.150591    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08e9108a-8670-47b9-a15b-a530895231e1-maas-api-xks-ca" pod "08e9108a-8670-47b9-a15b-a530895231e1" (UID: "08e9108a-8670-47b9-a15b-a530895231e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:34:29.150743 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:29.150719    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e9108a-8670-47b9-a15b-a530895231e1-kube-api-access-64shh" pod "08e9108a-8670-47b9-a15b-a530895231e1" (UID: "08e9108a-8670-47b9-a15b-a530895231e1"). InnerVolumeSpecName "kube-api-access-64shh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:34:29.218283 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:29.218226    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a3ef192-79tzf"]
Nov 03 21:34:29.218862 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:34:29.218838    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod243d8a52_c0be_4581_ab64_bf3897f88f1d.slice/crio-aff1668343209208dc04d7637fe2fa49202800051c9726bb5fe3f4579baba81d WatchSource:0}: Error finding container aff1668343209208dc04d7637fe2fa49202800051c9726bb5fe3f4579baba81d: Status 404 returned error can't find the container with id aff1668343209208dc04d7637fe2fa49202800051c9726bb5fe3f4579baba81d
Nov 03 21:34:29.250117 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:29.250079    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64shh\" (UniqueName: \"kubernetes.io/projected/08e9108a-8670-47b9-a15b-a530895231e1-kube-api-access-64shh\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:29.250117 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:29.250120    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08e9108a-8670-47b9-a15b-a530895231e1-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:29.250323 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:29.250135    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08e9108a-8670-47b9-a15b-a530895231e1-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:29.902121 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:29.902084    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ef192-79tzf" event={"ID":"243d8a52-c0be-4581-ab64-bf3897f88f1d","Type":"ContainerStarted","Data":"ca94d8d178f302425e637be85e28e140ec1366bd4fbd22358304a45e2c9b599c"}
Nov 03 21:34:29.902363 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:29.902162    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ef192-79tzf" event={"ID":"243d8a52-c0be-4581-ab64-bf3897f88f1d","Type":"ContainerStarted","Data":"aff1668343209208dc04d7637fe2fa49202800051c9726bb5fe3f4579baba81d"}
Nov 03 21:34:29.903752 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:29.903724    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb40886b-64qgl"
Nov 03 21:34:29.903897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:29.903799    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb40886b-64qgl" event={"ID":"08e9108a-8670-47b9-a15b-a530895231e1","Type":"ContainerDied","Data":"5215662b760d7228246d1e78ab62a1ecbda9d5ad2a46cbc394b21b3b02ca7869"}
Nov 03 21:34:29.903897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:29.903838    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5215662b760d7228246d1e78ab62a1ecbda9d5ad2a46cbc394b21b3b02ca7869"
Nov 03 21:34:30.022979 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:30.022920    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ef192-79tzf" podStartSLOduration=2.022898809 podStartE2EDuration="2.022898809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:30.016583952 +0000 UTC m=+1184.510505826" watchObservedRunningTime="2026-11-03 21:34:30.022898809 +0000 UTC m=+1184.516820684"
Nov 03 21:34:30.914697 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:30.914660    5053 generic.go:356] "Generic (PLEG): container finished" podID="243d8a52-c0be-4581-ab64-bf3897f88f1d" containerID="ca94d8d178f302425e637be85e28e140ec1366bd4fbd22358304a45e2c9b599c" exitCode=0
Nov 03 21:34:30.915115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:30.914746    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ef192-79tzf" event={"ID":"243d8a52-c0be-4581-ab64-bf3897f88f1d","Type":"ContainerDied","Data":"ca94d8d178f302425e637be85e28e140ec1366bd4fbd22358304a45e2c9b599c"}
Nov 03 21:34:32.379997 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:32.379970    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ef192-79tzf"
Nov 03 21:34:32.575698 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:32.575664    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/243d8a52-c0be-4581-ab64-bf3897f88f1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/243d8a52-c0be-4581-ab64-bf3897f88f1d-maas-api-openshift-ca\") pod \"243d8a52-c0be-4581-ab64-bf3897f88f1d\" (UID: \"243d8a52-c0be-4581-ab64-bf3897f88f1d\") "
Nov 03 21:34:32.575904 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:32.575719    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/243d8a52-c0be-4581-ab64-bf3897f88f1d-kube-api-access-xmq52\" (UniqueName: \"kubernetes.io/projected/243d8a52-c0be-4581-ab64-bf3897f88f1d-kube-api-access-xmq52\") pod \"243d8a52-c0be-4581-ab64-bf3897f88f1d\" (UID: \"243d8a52-c0be-4581-ab64-bf3897f88f1d\") "
Nov 03 21:34:32.575904 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:32.575879    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/243d8a52-c0be-4581-ab64-bf3897f88f1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/243d8a52-c0be-4581-ab64-bf3897f88f1d-maas-api-xks-ca\") pod \"243d8a52-c0be-4581-ab64-bf3897f88f1d\" (UID: \"243d8a52-c0be-4581-ab64-bf3897f88f1d\") "
Nov 03 21:34:32.577531 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:32.577089    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/243d8a52-c0be-4581-ab64-bf3897f88f1d-maas-api-openshift-ca" pod "243d8a52-c0be-4581-ab64-bf3897f88f1d" (UID: "243d8a52-c0be-4581-ab64-bf3897f88f1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:34:32.619416 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:32.619380    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/243d8a52-c0be-4581-ab64-bf3897f88f1d-maas-api-xks-ca" pod "243d8a52-c0be-4581-ab64-bf3897f88f1d" (UID: "243d8a52-c0be-4581-ab64-bf3897f88f1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:34:32.637286 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:32.635024    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/243d8a52-c0be-4581-ab64-bf3897f88f1d-kube-api-access-xmq52" pod "243d8a52-c0be-4581-ab64-bf3897f88f1d" (UID: "243d8a52-c0be-4581-ab64-bf3897f88f1d"). InnerVolumeSpecName "kube-api-access-xmq52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:34:32.680585 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:32.679030    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/243d8a52-c0be-4581-ab64-bf3897f88f1d-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:32.680585 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:32.679069    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmq52\" (UniqueName: \"kubernetes.io/projected/243d8a52-c0be-4581-ab64-bf3897f88f1d-kube-api-access-xmq52\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:32.680585 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:32.679088    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/243d8a52-c0be-4581-ab64-bf3897f88f1d-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:33.009270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:32.992077    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ef192-79tzf" event={"ID":"243d8a52-c0be-4581-ab64-bf3897f88f1d","Type":"ContainerDied","Data":"aff1668343209208dc04d7637fe2fa49202800051c9726bb5fe3f4579baba81d"}
Nov 03 21:34:33.009270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:32.992163    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aff1668343209208dc04d7637fe2fa49202800051c9726bb5fe3f4579baba81d"
Nov 03 21:34:33.009270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:32.992291    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ef192-79tzf"
Nov 03 21:34:33.943264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:33.942355    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8" podUID="43404f1c-c87c-4325-ab97-bffb6ea81e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Nov 03 21:34:34.013264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:34.011286    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx" podUID="a6c0d3aa-18c2-465b-846b-b82968acc780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:34:34.076263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:34.075355    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd" podUID="82077ea7-1e05-4b6e-806b-a2d2d2a64c4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:34:36.332225 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:36.332194    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23bb161c-sc4wg"]
Nov 03 21:34:36.333023 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:36.333004    5053 cpu_manager.go:414] "containerMap: removing container" podUID="243d8a52-c0be-4581-ab64-bf3897f88f1d" containerName="revoke-keys"
Nov 03 21:34:36.333131 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:36.333027    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="243d8a52-c0be-4581-ab64-bf3897f88f1d" containerName="revoke-keys"
Nov 03 21:34:36.333131 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:36.333080    5053 cpu_manager.go:414] "containerMap: removing container" podUID="08e9108a-8670-47b9-a15b-a530895231e1" containerName="revoke-keys"
Nov 03 21:34:36.333131 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:36.333089    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08e9108a-8670-47b9-a15b-a530895231e1" containerName="revoke-keys"
Nov 03 21:34:36.333332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:36.333269    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="08e9108a-8670-47b9-a15b-a530895231e1" containerName="revoke-keys"
Nov 03 21:34:36.333332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:36.333293    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="243d8a52-c0be-4581-ab64-bf3897f88f1d" containerName="revoke-keys"
Nov 03 21:34:36.337037 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:36.337017    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bb161c-sc4wg"
Nov 03 21:34:36.391564 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:36.391538    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23bb161c-sc4wg"]
Nov 03 21:34:36.436337 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:36.436308    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:34:36.436337 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:36.436330    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:34:36.436520 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:36.436330    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:34:36.438211 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:36.438191    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:34:36.445566 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:36.445544    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698a2a54-5236-49a9-b759-7659d7712b1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23bb161c-sc4wg\" (UID: \"698a2a54-5236-49a9-b759-7659d7712b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bb161c-sc4wg"
Nov 03 21:34:36.445677 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:36.445592    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698a2a54-5236-49a9-b759-7659d7712b1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23bb161c-sc4wg\" (UID: \"698a2a54-5236-49a9-b759-7659d7712b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bb161c-sc4wg"
Nov 03 21:34:36.445781 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:36.445758    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzjkk\" (UniqueName: \"kubernetes.io/projected/698a2a54-5236-49a9-b759-7659d7712b1c-kube-api-access-bzjkk\") pod \"maas-api-revoke-sub-23bb161c-sc4wg\" (UID: \"698a2a54-5236-49a9-b759-7659d7712b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bb161c-sc4wg"
Nov 03 21:34:36.546801 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:36.546763    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzjkk\" (UniqueName: \"kubernetes.io/projected/698a2a54-5236-49a9-b759-7659d7712b1c-kube-api-access-bzjkk\") pod \"maas-api-revoke-sub-23bb161c-sc4wg\" (UID: \"698a2a54-5236-49a9-b759-7659d7712b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bb161c-sc4wg"
Nov 03 21:34:36.546963 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:36.546932    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698a2a54-5236-49a9-b759-7659d7712b1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23bb161c-sc4wg\" (UID: \"698a2a54-5236-49a9-b759-7659d7712b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bb161c-sc4wg"
Nov 03 21:34:36.547032 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:36.546992    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698a2a54-5236-49a9-b759-7659d7712b1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23bb161c-sc4wg\" (UID: \"698a2a54-5236-49a9-b759-7659d7712b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bb161c-sc4wg"
Nov 03 21:34:36.547746 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:36.547722    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698a2a54-5236-49a9-b759-7659d7712b1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23bb161c-sc4wg\" (UID: \"698a2a54-5236-49a9-b759-7659d7712b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bb161c-sc4wg"
Nov 03 21:34:36.549665 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:36.549637    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698a2a54-5236-49a9-b759-7659d7712b1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23bb161c-sc4wg\" (UID: \"698a2a54-5236-49a9-b759-7659d7712b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bb161c-sc4wg"
Nov 03 21:34:36.603706 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:36.603626    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzjkk\" (UniqueName: \"kubernetes.io/projected/698a2a54-5236-49a9-b759-7659d7712b1c-kube-api-access-bzjkk\") pod \"maas-api-revoke-sub-23bb161c-sc4wg\" (UID: \"698a2a54-5236-49a9-b759-7659d7712b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bb161c-sc4wg"
Nov 03 21:34:36.654407 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:36.654371    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bb161c-sc4wg"
Nov 03 21:34:37.035196 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:34:37.035159    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698a2a54_5236_49a9_b759_7659d7712b1c.slice/crio-095b5ee0ac051f9bb5f13b23fbcc2f00a617f8ebe631a1df24b37d1755ddb101 WatchSource:0}: Error finding container 095b5ee0ac051f9bb5f13b23fbcc2f00a617f8ebe631a1df24b37d1755ddb101: Status 404 returned error can't find the container with id 095b5ee0ac051f9bb5f13b23fbcc2f00a617f8ebe631a1df24b37d1755ddb101
Nov 03 21:34:37.035376 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:37.035222    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23bb161c-sc4wg"]
Nov 03 21:34:37.046114 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:37.046083    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bb161c-sc4wg" event={"ID":"698a2a54-5236-49a9-b759-7659d7712b1c","Type":"ContainerStarted","Data":"095b5ee0ac051f9bb5f13b23fbcc2f00a617f8ebe631a1df24b37d1755ddb101"}
Nov 03 21:34:37.054322 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:37.054300    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa52af6-tmzrg"]
Nov 03 21:34:37.058194 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:37.058166    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa52af6-tmzrg"]
Nov 03 21:34:37.668138 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:37.668059    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e9dc7bc-a061-4404-84c3-23b3949232b0" path="/var/lib/kubelet/pods/9e9dc7bc-a061-4404-84c3-23b3949232b0/volumes"
Nov 03 21:34:38.055065 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:38.055033    5053 generic.go:356] "Generic (PLEG): container finished" podID="698a2a54-5236-49a9-b759-7659d7712b1c" containerID="1bc1c501c9ea9b6c6b5215adc2c7d760b2237126b7edb56d33332b60989474bb" exitCode=0
Nov 03 21:34:38.055065 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:38.055067    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bb161c-sc4wg" event={"ID":"698a2a54-5236-49a9-b759-7659d7712b1c","Type":"ContainerDied","Data":"1bc1c501c9ea9b6c6b5215adc2c7d760b2237126b7edb56d33332b60989474bb"}
Nov 03 21:34:39.207312 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:39.207284    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bb161c-sc4wg"
Nov 03 21:34:39.280822 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:39.280792    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/698a2a54-5236-49a9-b759-7659d7712b1c-kube-api-access-bzjkk\" (UniqueName: \"kubernetes.io/projected/698a2a54-5236-49a9-b759-7659d7712b1c-kube-api-access-bzjkk\") pod \"698a2a54-5236-49a9-b759-7659d7712b1c\" (UID: \"698a2a54-5236-49a9-b759-7659d7712b1c\") "
Nov 03 21:34:39.281140 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:39.280881    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/698a2a54-5236-49a9-b759-7659d7712b1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698a2a54-5236-49a9-b759-7659d7712b1c-maas-api-xks-ca\") pod \"698a2a54-5236-49a9-b759-7659d7712b1c\" (UID: \"698a2a54-5236-49a9-b759-7659d7712b1c\") "
Nov 03 21:34:39.281140 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:39.281028    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/698a2a54-5236-49a9-b759-7659d7712b1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698a2a54-5236-49a9-b759-7659d7712b1c-maas-api-openshift-ca\") pod \"698a2a54-5236-49a9-b759-7659d7712b1c\" (UID: \"698a2a54-5236-49a9-b759-7659d7712b1c\") "
Nov 03 21:34:39.281558 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:39.281532    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/698a2a54-5236-49a9-b759-7659d7712b1c-maas-api-openshift-ca" pod "698a2a54-5236-49a9-b759-7659d7712b1c" (UID: "698a2a54-5236-49a9-b759-7659d7712b1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:34:39.283426 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:39.283397    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/698a2a54-5236-49a9-b759-7659d7712b1c-maas-api-xks-ca" pod "698a2a54-5236-49a9-b759-7659d7712b1c" (UID: "698a2a54-5236-49a9-b759-7659d7712b1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:34:39.283726 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:39.283684    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698a2a54-5236-49a9-b759-7659d7712b1c-kube-api-access-bzjkk" pod "698a2a54-5236-49a9-b759-7659d7712b1c" (UID: "698a2a54-5236-49a9-b759-7659d7712b1c"). InnerVolumeSpecName "kube-api-access-bzjkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:34:39.284791 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:39.284766    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698a2a54-5236-49a9-b759-7659d7712b1c-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:39.284867 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:39.284799    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzjkk\" (UniqueName: \"kubernetes.io/projected/698a2a54-5236-49a9-b759-7659d7712b1c-kube-api-access-bzjkk\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:39.284867 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:39.284814    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698a2a54-5236-49a9-b759-7659d7712b1c-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:40.071314 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:40.071283    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bb161c-sc4wg"
Nov 03 21:34:40.071314 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:40.071291    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bb161c-sc4wg" event={"ID":"698a2a54-5236-49a9-b759-7659d7712b1c","Type":"ContainerDied","Data":"095b5ee0ac051f9bb5f13b23fbcc2f00a617f8ebe631a1df24b37d1755ddb101"}
Nov 03 21:34:40.071583 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:40.071338    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="095b5ee0ac051f9bb5f13b23fbcc2f00a617f8ebe631a1df24b37d1755ddb101"
Nov 03 21:34:40.928742 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:34:40.928706    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:34:45.841284 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:45.841255    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 21:34:45.850003 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:45.849980    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:34:45.854395 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:45.854376    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 21:34:45.861873 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:45.861850    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:34:45.875026 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:45.875004    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4a70c76-lqrt8"]
Nov 03 21:34:45.875655 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:45.875641    5053 cpu_manager.go:414] "containerMap: removing container" podUID="698a2a54-5236-49a9-b759-7659d7712b1c" containerName="revoke-keys"
Nov 03 21:34:45.875700 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:45.875657    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="698a2a54-5236-49a9-b759-7659d7712b1c" containerName="revoke-keys"
Nov 03 21:34:45.875777 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:45.875768    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="698a2a54-5236-49a9-b759-7659d7712b1c" containerName="revoke-keys"
Nov 03 21:34:45.880338 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:45.880322    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a70c76-lqrt8"
Nov 03 21:34:45.944808 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:45.944770    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4a70c76-lqrt8"]
Nov 03 21:34:45.968740 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:45.968707    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62eb6f28-db58-479d-9e7f-11d756f8fe0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4a70c76-lqrt8\" (UID: \"62eb6f28-db58-479d-9e7f-11d756f8fe0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a70c76-lqrt8"
Nov 03 21:34:45.968958 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:45.968936    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbqsn\" (UniqueName: \"kubernetes.io/projected/62eb6f28-db58-479d-9e7f-11d756f8fe0d-kube-api-access-wbqsn\") pod \"maas-api-revoke-sub-a4a70c76-lqrt8\" (UID: \"62eb6f28-db58-479d-9e7f-11d756f8fe0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a70c76-lqrt8"
Nov 03 21:34:45.969002 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:45.968968    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62eb6f28-db58-479d-9e7f-11d756f8fe0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4a70c76-lqrt8\" (UID: \"62eb6f28-db58-479d-9e7f-11d756f8fe0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a70c76-lqrt8"
Nov 03 21:34:45.980505 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:45.980475    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:34:45.980640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:45.980587    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:34:45.980640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:45.980617    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:34:45.980746 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:45.980657    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:34:46.070011 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:46.069980    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbqsn\" (UniqueName: \"kubernetes.io/projected/62eb6f28-db58-479d-9e7f-11d756f8fe0d-kube-api-access-wbqsn\") pod \"maas-api-revoke-sub-a4a70c76-lqrt8\" (UID: \"62eb6f28-db58-479d-9e7f-11d756f8fe0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a70c76-lqrt8"
Nov 03 21:34:46.070206 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:46.070018    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62eb6f28-db58-479d-9e7f-11d756f8fe0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4a70c76-lqrt8\" (UID: \"62eb6f28-db58-479d-9e7f-11d756f8fe0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a70c76-lqrt8"
Nov 03 21:34:46.070206 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:46.070123    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62eb6f28-db58-479d-9e7f-11d756f8fe0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4a70c76-lqrt8\" (UID: \"62eb6f28-db58-479d-9e7f-11d756f8fe0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a70c76-lqrt8"
Nov 03 21:34:46.070781 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:46.070716    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62eb6f28-db58-479d-9e7f-11d756f8fe0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4a70c76-lqrt8\" (UID: \"62eb6f28-db58-479d-9e7f-11d756f8fe0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a70c76-lqrt8"
Nov 03 21:34:46.072646 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:46.072619    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62eb6f28-db58-479d-9e7f-11d756f8fe0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4a70c76-lqrt8\" (UID: \"62eb6f28-db58-479d-9e7f-11d756f8fe0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a70c76-lqrt8"
Nov 03 21:34:46.126259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:46.126170    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbqsn\" (UniqueName: \"kubernetes.io/projected/62eb6f28-db58-479d-9e7f-11d756f8fe0d-kube-api-access-wbqsn\") pod \"maas-api-revoke-sub-a4a70c76-lqrt8\" (UID: \"62eb6f28-db58-479d-9e7f-11d756f8fe0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a70c76-lqrt8"
Nov 03 21:34:46.193310 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:46.193277    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a70c76-lqrt8"
Nov 03 21:34:46.578303 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:34:46.578272    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62eb6f28_db58_479d_9e7f_11d756f8fe0d.slice/crio-a60a0e05874262199b30b1987844f150e5ec86067c4dc05721008caea7f61928 WatchSource:0}: Error finding container a60a0e05874262199b30b1987844f150e5ec86067c4dc05721008caea7f61928: Status 404 returned error can't find the container with id a60a0e05874262199b30b1987844f150e5ec86067c4dc05721008caea7f61928
Nov 03 21:34:46.578711 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:46.578687    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4a70c76-lqrt8"]
Nov 03 21:34:47.164717 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:47.157739    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a70c76-lqrt8" event={"ID":"62eb6f28-db58-479d-9e7f-11d756f8fe0d","Type":"ContainerStarted","Data":"649ca2e9fb512cfd3068dfcd0147c5b67d4622812ccb01582733e3a3781370ac"}
Nov 03 21:34:47.164717 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:47.157806    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a70c76-lqrt8" event={"ID":"62eb6f28-db58-479d-9e7f-11d756f8fe0d","Type":"ContainerStarted","Data":"a60a0e05874262199b30b1987844f150e5ec86067c4dc05721008caea7f61928"}
Nov 03 21:34:48.280264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:48.263518    5053 generic.go:356] "Generic (PLEG): container finished" podID="62eb6f28-db58-479d-9e7f-11d756f8fe0d" containerID="649ca2e9fb512cfd3068dfcd0147c5b67d4622812ccb01582733e3a3781370ac" exitCode=0
Nov 03 21:34:48.280264 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:48.263761    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a70c76-lqrt8" event={"ID":"62eb6f28-db58-479d-9e7f-11d756f8fe0d","Type":"ContainerDied","Data":"649ca2e9fb512cfd3068dfcd0147c5b67d4622812ccb01582733e3a3781370ac"}
Nov 03 21:34:48.793261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:48.790360    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6" podUID="4803807f-9312-40ac-badb-00c00d3f3bd3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:34:49.027262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.016350    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx" podUID="a6c0d3aa-18c2-465b-846b-b82968acc780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:34:49.081315 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.080333    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd" podUID="82077ea7-1e05-4b6e-806b-a2d2d2a64c4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:34:49.180262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.179810    5053 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6" podUID="4803807f-9312-40ac-badb-00c00d3f3bd3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": context deadline exceeded"
Nov 03 21:34:49.436384 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.436283    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"]
Nov 03 21:34:49.500278 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.449855    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:49.628279 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.562581    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-envoy\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:49.629358 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.629319    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-workload-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:49.629621 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.629602    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-workload-certs\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:49.637465 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.637430    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfzk5\" (UniqueName: \"kubernetes.io/projected/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-kube-api-access-kfzk5\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:49.637660 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.637637    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-data\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:49.643268 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.637771    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-token\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:49.643502 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.643471    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istiod-ca-cert\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:49.647549 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.647520    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-credential-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:49.654342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.654309    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-podinfo\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:49.793413 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.793368    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6" podUID="4803807f-9312-40ac-badb-00c00d3f3bd3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:34:49.850217 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.850174    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfzk5\" (UniqueName: \"kubernetes.io/projected/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-kube-api-access-kfzk5\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:49.850528 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.850503    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-data\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:49.850818 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.850797    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-token\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:49.850949 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.850932    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istiod-ca-cert\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:49.851131 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.851112    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-credential-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:49.851500 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.851269    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-podinfo\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:49.851682 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.851662    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-envoy\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:49.851874 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.851854    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-workload-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:49.852086 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.852068    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-workload-certs\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:49.871893 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.871860    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-workload-certs\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:49.885460 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.885428    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-data\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:49.912263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.886933    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istiod-ca-cert\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:49.978961 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:49.955938    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-credential-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:50.246300 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.239083    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n" podUID="512b3ab1-45ba-4f96-9799-b129b25fca42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:34:50.285617 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.245458    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8" podUID="43404f1c-c87c-4325-ab97-bffb6ea81e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Nov 03 21:34:50.287270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.245500    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx" podUID="a6c0d3aa-18c2-465b-846b-b82968acc780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 03 21:34:50.287916 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.245554    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd" podUID="82077ea7-1e05-4b6e-806b-a2d2d2a64c4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 21:34:50.289067 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.284171    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfzk5\" (UniqueName: \"kubernetes.io/projected/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-kube-api-access-kfzk5\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:50.495267 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.485438    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-envoy\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:50.539392 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.538379    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-workload-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:50.546711 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.546680    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-token\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:50.549787 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.549762    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:34:50.549997 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.549982    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:34:50.550120 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.550107    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"]
Nov 03 21:34:50.550286 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.550272    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:34:50.562132 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.561996    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:34:50.579505 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.579199    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-podinfo\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:50.590882 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.590853    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:34:50.591823 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.591789    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a70c76-lqrt8" event={"ID":"62eb6f28-db58-479d-9e7f-11d756f8fe0d","Type":"ContainerDied","Data":"a60a0e05874262199b30b1987844f150e5ec86067c4dc05721008caea7f61928"}
Nov 03 21:34:50.591954 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.591848    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a60a0e05874262199b30b1987844f150e5ec86067c4dc05721008caea7f61928"
Nov 03 21:34:50.593876 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.593789    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a70c76-lqrt8"
Nov 03 21:34:50.612041 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.611627    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:34:50.761567 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.761524    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:50.766095 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.766050    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62eb6f28-db58-479d-9e7f-11d756f8fe0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62eb6f28-db58-479d-9e7f-11d756f8fe0d-maas-api-openshift-ca\") pod \"62eb6f28-db58-479d-9e7f-11d756f8fe0d\" (UID: \"62eb6f28-db58-479d-9e7f-11d756f8fe0d\") "
Nov 03 21:34:50.766214 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.766200    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62eb6f28-db58-479d-9e7f-11d756f8fe0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62eb6f28-db58-479d-9e7f-11d756f8fe0d-maas-api-xks-ca\") pod \"62eb6f28-db58-479d-9e7f-11d756f8fe0d\" (UID: \"62eb6f28-db58-479d-9e7f-11d756f8fe0d\") "
Nov 03 21:34:50.766331 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.766286    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62eb6f28-db58-479d-9e7f-11d756f8fe0d-kube-api-access-wbqsn\" (UniqueName: \"kubernetes.io/projected/62eb6f28-db58-479d-9e7f-11d756f8fe0d-kube-api-access-wbqsn\") pod \"62eb6f28-db58-479d-9e7f-11d756f8fe0d\" (UID: \"62eb6f28-db58-479d-9e7f-11d756f8fe0d\") "
Nov 03 21:34:50.766515 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.766482    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62eb6f28-db58-479d-9e7f-11d756f8fe0d-maas-api-openshift-ca" pod "62eb6f28-db58-479d-9e7f-11d756f8fe0d" (UID: "62eb6f28-db58-479d-9e7f-11d756f8fe0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:34:50.768006 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.767969    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62eb6f28-db58-479d-9e7f-11d756f8fe0d-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:50.769164 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.769138    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62eb6f28-db58-479d-9e7f-11d756f8fe0d-kube-api-access-wbqsn" pod "62eb6f28-db58-479d-9e7f-11d756f8fe0d" (UID: "62eb6f28-db58-479d-9e7f-11d756f8fe0d"). InnerVolumeSpecName "kube-api-access-wbqsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:34:50.769371 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.769269    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62eb6f28-db58-479d-9e7f-11d756f8fe0d-maas-api-xks-ca" pod "62eb6f28-db58-479d-9e7f-11d756f8fe0d" (UID: "62eb6f28-db58-479d-9e7f-11d756f8fe0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:34:50.869173 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.868988    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62eb6f28-db58-479d-9e7f-11d756f8fe0d-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:50.869173 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:50.869023    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbqsn\" (UniqueName: \"kubernetes.io/projected/62eb6f28-db58-479d-9e7f-11d756f8fe0d-kube-api-access-wbqsn\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:51.222498 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:34:51.222463    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1d9743_441e_4e3a_a42b_c521bd99b7c3.slice/crio-7832b670a4fd109acd3aa64d8d7b70e96ccc9e9d5863d4b73b69e27ed4657424 WatchSource:0}: Error finding container 7832b670a4fd109acd3aa64d8d7b70e96ccc9e9d5863d4b73b69e27ed4657424: Status 404 returned error can't find the container with id 7832b670a4fd109acd3aa64d8d7b70e96ccc9e9d5863d4b73b69e27ed4657424
Nov 03 21:34:51.224870 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:51.224839    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"]
Nov 03 21:34:51.225652 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:51.225619    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:34:51.225712 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:51.225697    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:34:51.225753 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:51.225736    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:34:51.605895 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:51.605859    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6" event={"ID":"7b1d9743-441e-4e3a-a42b-c521bd99b7c3","Type":"ContainerStarted","Data":"c347183848d9e24f307629b3a84b05c9e6a9140a6a2bbfc77b5df89587e71004"}
Nov 03 21:34:51.606398 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:51.605917    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6" event={"ID":"7b1d9743-441e-4e3a-a42b-c521bd99b7c3","Type":"ContainerStarted","Data":"7832b670a4fd109acd3aa64d8d7b70e96ccc9e9d5863d4b73b69e27ed4657424"}
Nov 03 21:34:51.606398 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:51.606330    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a70c76-lqrt8"
Nov 03 21:34:51.724278 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:51.722980    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6" podStartSLOduration=2.722961133 podStartE2EDuration="2.722961133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:51.722340819 +0000 UTC m=+1206.216262694" watchObservedRunningTime="2026-11-03 21:34:51.722961133 +0000 UTC m=+1206.216883008"
Nov 03 21:34:51.762115 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:51.762077    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:52.763675 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:52.763638    5053 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 21:34:52.764069 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:52.763693    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6" podUID="7b1d9743-441e-4e3a-a42b-c521bd99b7c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 21:34:52.956807 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:52.956770    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9e7ee1-dbdgz"]
Nov 03 21:34:52.957417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:52.957399    5053 cpu_manager.go:414] "containerMap: removing container" podUID="62eb6f28-db58-479d-9e7f-11d756f8fe0d" containerName="revoke-keys"
Nov 03 21:34:52.957417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:52.957414    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62eb6f28-db58-479d-9e7f-11d756f8fe0d" containerName="revoke-keys"
Nov 03 21:34:52.957620 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:52.957557    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="62eb6f28-db58-479d-9e7f-11d756f8fe0d" containerName="revoke-keys"
Nov 03 21:34:52.961450 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:52.961428    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9e7ee1-dbdgz"
Nov 03 21:34:52.992514 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:52.992481    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89850b47-7092-4eed-a237-72cc22796def-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a9e7ee1-dbdgz\" (UID: \"89850b47-7092-4eed-a237-72cc22796def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9e7ee1-dbdgz"
Nov 03 21:34:52.992672 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:52.992534    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89850b47-7092-4eed-a237-72cc22796def-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a9e7ee1-dbdgz\" (UID: \"89850b47-7092-4eed-a237-72cc22796def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9e7ee1-dbdgz"
Nov 03 21:34:52.992672 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:52.992649    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqgdc\" (UniqueName: \"kubernetes.io/projected/89850b47-7092-4eed-a237-72cc22796def-kube-api-access-jqgdc\") pod \"maas-api-revoke-sub-2a9e7ee1-dbdgz\" (UID: \"89850b47-7092-4eed-a237-72cc22796def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9e7ee1-dbdgz"
Nov 03 21:34:53.043289 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:53.043079    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9e7ee1-dbdgz"]
Nov 03 21:34:53.066396 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:53.066356    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:34:53.066396 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:53.066385    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:34:53.066607 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:53.066390    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:34:53.066607 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:53.066414    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:34:53.094544 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:53.094515    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89850b47-7092-4eed-a237-72cc22796def-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a9e7ee1-dbdgz\" (UID: \"89850b47-7092-4eed-a237-72cc22796def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9e7ee1-dbdgz"
Nov 03 21:34:53.094715 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:53.094582    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89850b47-7092-4eed-a237-72cc22796def-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a9e7ee1-dbdgz\" (UID: \"89850b47-7092-4eed-a237-72cc22796def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9e7ee1-dbdgz"
Nov 03 21:34:53.094715 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:53.094646    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqgdc\" (UniqueName: \"kubernetes.io/projected/89850b47-7092-4eed-a237-72cc22796def-kube-api-access-jqgdc\") pod \"maas-api-revoke-sub-2a9e7ee1-dbdgz\" (UID: \"89850b47-7092-4eed-a237-72cc22796def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9e7ee1-dbdgz"
Nov 03 21:34:53.095181 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:53.095162    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89850b47-7092-4eed-a237-72cc22796def-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a9e7ee1-dbdgz\" (UID: \"89850b47-7092-4eed-a237-72cc22796def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9e7ee1-dbdgz"
Nov 03 21:34:53.097146 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:53.097125    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89850b47-7092-4eed-a237-72cc22796def-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a9e7ee1-dbdgz\" (UID: \"89850b47-7092-4eed-a237-72cc22796def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9e7ee1-dbdgz"
Nov 03 21:34:53.151484 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:53.151442    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqgdc\" (UniqueName: \"kubernetes.io/projected/89850b47-7092-4eed-a237-72cc22796def-kube-api-access-jqgdc\") pod \"maas-api-revoke-sub-2a9e7ee1-dbdgz\" (UID: \"89850b47-7092-4eed-a237-72cc22796def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9e7ee1-dbdgz"
Nov 03 21:34:53.276122 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:53.276092    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9e7ee1-dbdgz"
Nov 03 21:34:53.688741 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:53.688706    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9e7ee1-dbdgz"]
Nov 03 21:34:53.762495 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:53.762464    5053 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 21:34:53.762623 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:53.762522    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6" podUID="7b1d9743-441e-4e3a-a42b-c521bd99b7c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 21:34:54.652597 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:54.652336    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9e7ee1-dbdgz" event={"ID":"89850b47-7092-4eed-a237-72cc22796def","Type":"ContainerStarted","Data":"2358a68f0ab7edfa29098ac5719e00114e2e1b6915ff6f0b7df349b906d25c35"}
Nov 03 21:34:54.652597 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:54.652394    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9e7ee1-dbdgz" event={"ID":"89850b47-7092-4eed-a237-72cc22796def","Type":"ContainerStarted","Data":"8308290a961be534b2457b85c518b2a2739c0c7a3a941e397d75f7e5338c00e7"}
Nov 03 21:34:54.762041 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:54.762014    5053 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 21:34:54.762181 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:54.762066    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6" podUID="7b1d9743-441e-4e3a-a42b-c521bd99b7c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 21:34:54.771407 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:54.771346    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9e7ee1-dbdgz" podStartSLOduration=2.771332056 podStartE2EDuration="2.771332056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:54.768144239 +0000 UTC m=+1209.262066112" watchObservedRunningTime="2026-11-03 21:34:54.771332056 +0000 UTC m=+1209.265253979"
Nov 03 21:34:55.046919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:55.046891    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f5a00ef-kpnvh"]
Nov 03 21:34:55.050893 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:55.050870    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f5a00ef-kpnvh"]
Nov 03 21:34:55.662074 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:55.662038    5053 generic.go:356] "Generic (PLEG): container finished" podID="89850b47-7092-4eed-a237-72cc22796def" containerID="2358a68f0ab7edfa29098ac5719e00114e2e1b6915ff6f0b7df349b906d25c35" exitCode=0
Nov 03 21:34:55.671474 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:55.671443    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="309804de-c1da-49fc-96b3-53ba8be575fa" path="/var/lib/kubelet/pods/309804de-c1da-49fc-96b3-53ba8be575fa/volumes"
Nov 03 21:34:55.671835 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:55.671812    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9e7ee1-dbdgz" event={"ID":"89850b47-7092-4eed-a237-72cc22796def","Type":"ContainerDied","Data":"2358a68f0ab7edfa29098ac5719e00114e2e1b6915ff6f0b7df349b906d25c35"}
Nov 03 21:34:55.762486 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:55.762450    5053 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 21:34:55.762702 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:55.762514    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6" podUID="7b1d9743-441e-4e3a-a42b-c521bd99b7c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 21:34:56.640884 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:56.640787    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b33a1c-lmj2b"]
Nov 03 21:34:56.645713 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:56.645684    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b33a1c-lmj2b"
Nov 03 21:34:56.686970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:56.686938    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmgfj\" (UniqueName: \"kubernetes.io/projected/e4f15d8b-0518-4a43-b22f-508457222b3b-kube-api-access-pmgfj\") pod \"maas-api-revoke-sub-b6b33a1c-lmj2b\" (UID: \"e4f15d8b-0518-4a43-b22f-508457222b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b33a1c-lmj2b"
Nov 03 21:34:56.687438 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:56.687060    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f15d8b-0518-4a43-b22f-508457222b3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6b33a1c-lmj2b\" (UID: \"e4f15d8b-0518-4a43-b22f-508457222b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b33a1c-lmj2b"
Nov 03 21:34:56.687438 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:56.687150    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f15d8b-0518-4a43-b22f-508457222b3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6b33a1c-lmj2b\" (UID: \"e4f15d8b-0518-4a43-b22f-508457222b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b33a1c-lmj2b"
Nov 03 21:34:56.721268 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:56.712554    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b33a1c-lmj2b"]
Nov 03 21:34:56.762005 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:56.761948    5053 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 21:34:56.762107 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:56.762038    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6" podUID="7b1d9743-441e-4e3a-a42b-c521bd99b7c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 21:34:56.789450 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:56.789378    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmgfj\" (UniqueName: \"kubernetes.io/projected/e4f15d8b-0518-4a43-b22f-508457222b3b-kube-api-access-pmgfj\") pod \"maas-api-revoke-sub-b6b33a1c-lmj2b\" (UID: \"e4f15d8b-0518-4a43-b22f-508457222b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b33a1c-lmj2b"
Nov 03 21:34:56.789684 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:56.789478    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f15d8b-0518-4a43-b22f-508457222b3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6b33a1c-lmj2b\" (UID: \"e4f15d8b-0518-4a43-b22f-508457222b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b33a1c-lmj2b"
Nov 03 21:34:56.789684 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:56.789536    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f15d8b-0518-4a43-b22f-508457222b3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6b33a1c-lmj2b\" (UID: \"e4f15d8b-0518-4a43-b22f-508457222b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b33a1c-lmj2b"
Nov 03 21:34:56.791101 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:56.790966    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f15d8b-0518-4a43-b22f-508457222b3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6b33a1c-lmj2b\" (UID: \"e4f15d8b-0518-4a43-b22f-508457222b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b33a1c-lmj2b"
Nov 03 21:34:56.830707 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:56.829830    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f15d8b-0518-4a43-b22f-508457222b3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6b33a1c-lmj2b\" (UID: \"e4f15d8b-0518-4a43-b22f-508457222b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b33a1c-lmj2b"
Nov 03 21:34:56.848197 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:56.848135    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmgfj\" (UniqueName: \"kubernetes.io/projected/e4f15d8b-0518-4a43-b22f-508457222b3b-kube-api-access-pmgfj\") pod \"maas-api-revoke-sub-b6b33a1c-lmj2b\" (UID: \"e4f15d8b-0518-4a43-b22f-508457222b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b33a1c-lmj2b"
Nov 03 21:34:56.929523 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:56.929448    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9e7ee1-dbdgz"
Nov 03 21:34:56.960353 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:56.960325    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b33a1c-lmj2b"
Nov 03 21:34:56.991642 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:56.991616    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89850b47-7092-4eed-a237-72cc22796def-kube-api-access-jqgdc\" (UniqueName: \"kubernetes.io/projected/89850b47-7092-4eed-a237-72cc22796def-kube-api-access-jqgdc\") pod \"89850b47-7092-4eed-a237-72cc22796def\" (UID: \"89850b47-7092-4eed-a237-72cc22796def\") "
Nov 03 21:34:56.991789 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:56.991663    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89850b47-7092-4eed-a237-72cc22796def-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89850b47-7092-4eed-a237-72cc22796def-maas-api-xks-ca\") pod \"89850b47-7092-4eed-a237-72cc22796def\" (UID: \"89850b47-7092-4eed-a237-72cc22796def\") "
Nov 03 21:34:56.991789 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:56.991702    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89850b47-7092-4eed-a237-72cc22796def-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89850b47-7092-4eed-a237-72cc22796def-maas-api-openshift-ca\") pod \"89850b47-7092-4eed-a237-72cc22796def\" (UID: \"89850b47-7092-4eed-a237-72cc22796def\") "
Nov 03 21:34:56.992167 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:56.992130    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89850b47-7092-4eed-a237-72cc22796def-maas-api-openshift-ca" pod "89850b47-7092-4eed-a237-72cc22796def" (UID: "89850b47-7092-4eed-a237-72cc22796def"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:34:56.992492 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:56.992471    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89850b47-7092-4eed-a237-72cc22796def-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:56.993869 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:56.993847    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89850b47-7092-4eed-a237-72cc22796def-maas-api-xks-ca" pod "89850b47-7092-4eed-a237-72cc22796def" (UID: "89850b47-7092-4eed-a237-72cc22796def"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:34:56.993974 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:56.993939    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89850b47-7092-4eed-a237-72cc22796def-kube-api-access-jqgdc" pod "89850b47-7092-4eed-a237-72cc22796def" (UID: "89850b47-7092-4eed-a237-72cc22796def"). InnerVolumeSpecName "kube-api-access-jqgdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:34:57.093284 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:57.093218    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqgdc\" (UniqueName: \"kubernetes.io/projected/89850b47-7092-4eed-a237-72cc22796def-kube-api-access-jqgdc\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:57.093284 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:57.093280    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89850b47-7092-4eed-a237-72cc22796def-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:57.361549 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:34:57.361518    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f15d8b_0518_4a43_b22f_508457222b3b.slice/crio-87036aa2c3e7fba539313cbf63a4b081088d84c9aab577d4fe62a167a7914c73 WatchSource:0}: Error finding container 87036aa2c3e7fba539313cbf63a4b081088d84c9aab577d4fe62a167a7914c73: Status 404 returned error can't find the container with id 87036aa2c3e7fba539313cbf63a4b081088d84c9aab577d4fe62a167a7914c73
Nov 03 21:34:57.361720 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:57.361647    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b33a1c-lmj2b"]
Nov 03 21:34:57.683509 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:57.683486    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9e7ee1-dbdgz"
Nov 03 21:34:57.683653 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:57.683535    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9e7ee1-dbdgz" event={"ID":"89850b47-7092-4eed-a237-72cc22796def","Type":"ContainerDied","Data":"8308290a961be534b2457b85c518b2a2739c0c7a3a941e397d75f7e5338c00e7"}
Nov 03 21:34:57.683653 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:57.683595    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8308290a961be534b2457b85c518b2a2739c0c7a3a941e397d75f7e5338c00e7"
Nov 03 21:34:57.685717 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:57.685652    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b33a1c-lmj2b" event={"ID":"e4f15d8b-0518-4a43-b22f-508457222b3b","Type":"ContainerStarted","Data":"6cd99b67dc382acd3ed6ed706f1feaeff873ea7b03b527036e91188b19ff42ac"}
Nov 03 21:34:57.685866 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:57.685741    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b33a1c-lmj2b" event={"ID":"e4f15d8b-0518-4a43-b22f-508457222b3b","Type":"ContainerStarted","Data":"87036aa2c3e7fba539313cbf63a4b081088d84c9aab577d4fe62a167a7914c73"}
Nov 03 21:34:57.762746 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:57.762713    5053 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 21:34:57.763125 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:57.762768    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6" podUID="7b1d9743-441e-4e3a-a42b-c521bd99b7c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 21:34:57.801617 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:57.801562    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b33a1c-lmj2b" podStartSLOduration=1.801547985 podStartE2EDuration="1.801547985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:57.798100965 +0000 UTC m=+1212.292022855" watchObservedRunningTime="2026-11-03 21:34:57.801547985 +0000 UTC m=+1212.295469858"
Nov 03 21:34:58.694659 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:58.694626    5053 generic.go:356] "Generic (PLEG): container finished" podID="e4f15d8b-0518-4a43-b22f-508457222b3b" containerID="6cd99b67dc382acd3ed6ed706f1feaeff873ea7b03b527036e91188b19ff42ac" exitCode=0
Nov 03 21:34:58.694849 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:58.694705    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b33a1c-lmj2b" event={"ID":"e4f15d8b-0518-4a43-b22f-508457222b3b","Type":"ContainerDied","Data":"6cd99b67dc382acd3ed6ed706f1feaeff873ea7b03b527036e91188b19ff42ac"}
Nov 03 21:34:58.766357 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:58.766325    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:58.766800 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:58.766622    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:58.767420 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:58.767403    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:34:59.868913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:59.868886    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b33a1c-lmj2b"
Nov 03 21:34:59.932764 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:59.932731    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4f15d8b-0518-4a43-b22f-508457222b3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f15d8b-0518-4a43-b22f-508457222b3b-maas-api-openshift-ca\") pod \"e4f15d8b-0518-4a43-b22f-508457222b3b\" (UID: \"e4f15d8b-0518-4a43-b22f-508457222b3b\") "
Nov 03 21:34:59.932967 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:59.932839    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4f15d8b-0518-4a43-b22f-508457222b3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f15d8b-0518-4a43-b22f-508457222b3b-maas-api-xks-ca\") pod \"e4f15d8b-0518-4a43-b22f-508457222b3b\" (UID: \"e4f15d8b-0518-4a43-b22f-508457222b3b\") "
Nov 03 21:34:59.932967 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:59.932907    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4f15d8b-0518-4a43-b22f-508457222b3b-kube-api-access-pmgfj\" (UniqueName: \"kubernetes.io/projected/e4f15d8b-0518-4a43-b22f-508457222b3b-kube-api-access-pmgfj\") pod \"e4f15d8b-0518-4a43-b22f-508457222b3b\" (UID: \"e4f15d8b-0518-4a43-b22f-508457222b3b\") "
Nov 03 21:34:59.933187 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:59.933160    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f15d8b-0518-4a43-b22f-508457222b3b-maas-api-openshift-ca" pod "e4f15d8b-0518-4a43-b22f-508457222b3b" (UID: "e4f15d8b-0518-4a43-b22f-508457222b3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:34:59.933542 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:59.933486    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f15d8b-0518-4a43-b22f-508457222b3b-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:59.935742 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:59.935706    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4f15d8b-0518-4a43-b22f-508457222b3b-maas-api-xks-ca" pod "e4f15d8b-0518-4a43-b22f-508457222b3b" (UID: "e4f15d8b-0518-4a43-b22f-508457222b3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:34:59.935877 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:34:59.935853    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f15d8b-0518-4a43-b22f-508457222b3b-kube-api-access-pmgfj" pod "e4f15d8b-0518-4a43-b22f-508457222b3b" (UID: "e4f15d8b-0518-4a43-b22f-508457222b3b"). InnerVolumeSpecName "kube-api-access-pmgfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:35:00.035120 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:00.035016    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmgfj\" (UniqueName: \"kubernetes.io/projected/e4f15d8b-0518-4a43-b22f-508457222b3b-kube-api-access-pmgfj\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:35:00.035120 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:00.035057    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f15d8b-0518-4a43-b22f-508457222b3b-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:35:00.712045 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:00.712010    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b33a1c-lmj2b" event={"ID":"e4f15d8b-0518-4a43-b22f-508457222b3b","Type":"ContainerDied","Data":"87036aa2c3e7fba539313cbf63a4b081088d84c9aab577d4fe62a167a7914c73"}
Nov 03 21:35:00.712045 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:00.712045    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b33a1c-lmj2b"
Nov 03 21:35:00.712296 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:00.712067    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87036aa2c3e7fba539313cbf63a4b081088d84c9aab577d4fe62a167a7914c73"
Nov 03 21:35:03.708031 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:03.707996    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f8f073b-94nrl"]
Nov 03 21:35:03.708641 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:03.708627    5053 cpu_manager.go:414] "containerMap: removing container" podUID="89850b47-7092-4eed-a237-72cc22796def" containerName="revoke-keys"
Nov 03 21:35:03.708730 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:03.708642    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89850b47-7092-4eed-a237-72cc22796def" containerName="revoke-keys"
Nov 03 21:35:03.708730 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:03.708660    5053 cpu_manager.go:414] "containerMap: removing container" podUID="e4f15d8b-0518-4a43-b22f-508457222b3b" containerName="revoke-keys"
Nov 03 21:35:03.708730 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:03.708665    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f15d8b-0518-4a43-b22f-508457222b3b" containerName="revoke-keys"
Nov 03 21:35:03.708820 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:03.708749    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="89850b47-7092-4eed-a237-72cc22796def" containerName="revoke-keys"
Nov 03 21:35:03.708820 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:03.708758    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f15d8b-0518-4a43-b22f-508457222b3b" containerName="revoke-keys"
Nov 03 21:35:03.712565 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:03.712546    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8f073b-94nrl"
Nov 03 21:35:03.764558 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:03.764521    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:35:03.767905 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:03.767864    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f8f073b-94nrl"]
Nov 03 21:35:03.778413 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:03.778377    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253e3305-deb7-4478-844b-387347d69de0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f8f073b-94nrl\" (UID: \"253e3305-deb7-4478-844b-387347d69de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8f073b-94nrl"
Nov 03 21:35:03.778974 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:03.778622    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j49qm\" (UniqueName: \"kubernetes.io/projected/253e3305-deb7-4478-844b-387347d69de0-kube-api-access-j49qm\") pod \"maas-api-revoke-sub-2f8f073b-94nrl\" (UID: \"253e3305-deb7-4478-844b-387347d69de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8f073b-94nrl"
Nov 03 21:35:03.778974 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:03.778714    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253e3305-deb7-4478-844b-387347d69de0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f8f073b-94nrl\" (UID: \"253e3305-deb7-4478-844b-387347d69de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8f073b-94nrl"
Nov 03 21:35:03.812220 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:03.812182    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:35:03.812453 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:03.812384    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:35:03.812453 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:03.812422    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:35:03.880446 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:03.880410    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253e3305-deb7-4478-844b-387347d69de0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f8f073b-94nrl\" (UID: \"253e3305-deb7-4478-844b-387347d69de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8f073b-94nrl"
Nov 03 21:35:03.880625 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:03.880508    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253e3305-deb7-4478-844b-387347d69de0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f8f073b-94nrl\" (UID: \"253e3305-deb7-4478-844b-387347d69de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8f073b-94nrl"
Nov 03 21:35:03.880625 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:03.880619    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j49qm\" (UniqueName: \"kubernetes.io/projected/253e3305-deb7-4478-844b-387347d69de0-kube-api-access-j49qm\") pod \"maas-api-revoke-sub-2f8f073b-94nrl\" (UID: \"253e3305-deb7-4478-844b-387347d69de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8f073b-94nrl"
Nov 03 21:35:03.881211 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:03.881191    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253e3305-deb7-4478-844b-387347d69de0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f8f073b-94nrl\" (UID: \"253e3305-deb7-4478-844b-387347d69de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8f073b-94nrl"
Nov 03 21:35:03.883168 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:03.883143    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253e3305-deb7-4478-844b-387347d69de0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f8f073b-94nrl\" (UID: \"253e3305-deb7-4478-844b-387347d69de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8f073b-94nrl"
Nov 03 21:35:03.950787 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:03.950748    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j49qm\" (UniqueName: \"kubernetes.io/projected/253e3305-deb7-4478-844b-387347d69de0-kube-api-access-j49qm\") pod \"maas-api-revoke-sub-2f8f073b-94nrl\" (UID: \"253e3305-deb7-4478-844b-387347d69de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8f073b-94nrl"
Nov 03 21:35:04.029692 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:04.029596    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8f073b-94nrl"
Nov 03 21:35:04.408970 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:35:04.408934    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253e3305_deb7_4478_844b_387347d69de0.slice/crio-92ce8b61b97c441cc25acf9bf5e38b8aa1ae2db614167f446e9996548146f4a4 WatchSource:0}: Error finding container 92ce8b61b97c441cc25acf9bf5e38b8aa1ae2db614167f446e9996548146f4a4: Status 404 returned error can't find the container with id 92ce8b61b97c441cc25acf9bf5e38b8aa1ae2db614167f446e9996548146f4a4
Nov 03 21:35:04.409470 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:04.409449    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f8f073b-94nrl"]
Nov 03 21:35:04.746201 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:04.746170    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8f073b-94nrl" event={"ID":"253e3305-deb7-4478-844b-387347d69de0","Type":"ContainerStarted","Data":"44d28cd51932c648c9fd7a50a7c1bbdcf632b5d73c07edfd9050a4012144163d"}
Nov 03 21:35:04.746701 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:04.746228    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8f073b-94nrl" event={"ID":"253e3305-deb7-4478-844b-387347d69de0","Type":"ContainerStarted","Data":"92ce8b61b97c441cc25acf9bf5e38b8aa1ae2db614167f446e9996548146f4a4"}
Nov 03 21:35:04.861896 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:04.861842    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8f073b-94nrl" podStartSLOduration=1.861828287 podStartE2EDuration="1.861828287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:04.858309501 +0000 UTC m=+1219.352231378" watchObservedRunningTime="2026-11-03 21:35:04.861828287 +0000 UTC m=+1219.355750161"
Nov 03 21:35:05.756250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:05.756199    5053 generic.go:356] "Generic (PLEG): container finished" podID="253e3305-deb7-4478-844b-387347d69de0" containerID="44d28cd51932c648c9fd7a50a7c1bbdcf632b5d73c07edfd9050a4012144163d" exitCode=0
Nov 03 21:35:05.756659 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:05.756271    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8f073b-94nrl" event={"ID":"253e3305-deb7-4478-844b-387347d69de0","Type":"ContainerDied","Data":"44d28cd51932c648c9fd7a50a7c1bbdcf632b5d73c07edfd9050a4012144163d"}
Nov 03 21:35:06.918609 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:06.918561    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8f073b-94nrl"
Nov 03 21:35:07.020410 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:07.020326    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/253e3305-deb7-4478-844b-387347d69de0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253e3305-deb7-4478-844b-387347d69de0-maas-api-xks-ca\") pod \"253e3305-deb7-4478-844b-387347d69de0\" (UID: \"253e3305-deb7-4478-844b-387347d69de0\") "
Nov 03 21:35:07.020637 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:07.020617    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/253e3305-deb7-4478-844b-387347d69de0-kube-api-access-j49qm\" (UniqueName: \"kubernetes.io/projected/253e3305-deb7-4478-844b-387347d69de0-kube-api-access-j49qm\") pod \"253e3305-deb7-4478-844b-387347d69de0\" (UID: \"253e3305-deb7-4478-844b-387347d69de0\") "
Nov 03 21:35:07.020686 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:07.020672    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/253e3305-deb7-4478-844b-387347d69de0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253e3305-deb7-4478-844b-387347d69de0-maas-api-openshift-ca\") pod \"253e3305-deb7-4478-844b-387347d69de0\" (UID: \"253e3305-deb7-4478-844b-387347d69de0\") "
Nov 03 21:35:07.021098 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:07.021068    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/253e3305-deb7-4478-844b-387347d69de0-maas-api-openshift-ca" pod "253e3305-deb7-4478-844b-387347d69de0" (UID: "253e3305-deb7-4478-844b-387347d69de0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:35:07.021283 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:07.021195    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253e3305-deb7-4478-844b-387347d69de0-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:35:07.022794 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:07.022765    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/253e3305-deb7-4478-844b-387347d69de0-maas-api-xks-ca" pod "253e3305-deb7-4478-844b-387347d69de0" (UID: "253e3305-deb7-4478-844b-387347d69de0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:35:07.023217 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:07.023191    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253e3305-deb7-4478-844b-387347d69de0-kube-api-access-j49qm" pod "253e3305-deb7-4478-844b-387347d69de0" (UID: "253e3305-deb7-4478-844b-387347d69de0"). InnerVolumeSpecName "kube-api-access-j49qm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:35:07.122172 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:07.122130    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j49qm\" (UniqueName: \"kubernetes.io/projected/253e3305-deb7-4478-844b-387347d69de0-kube-api-access-j49qm\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:35:07.122172 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:07.122167    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253e3305-deb7-4478-844b-387347d69de0-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:35:07.772491 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:07.772465    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8f073b-94nrl"
Nov 03 21:35:07.772667 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:07.772501    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f8f073b-94nrl" event={"ID":"253e3305-deb7-4478-844b-387347d69de0","Type":"ContainerDied","Data":"92ce8b61b97c441cc25acf9bf5e38b8aa1ae2db614167f446e9996548146f4a4"}
Nov 03 21:35:07.772667 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:07.772547    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92ce8b61b97c441cc25acf9bf5e38b8aa1ae2db614167f446e9996548146f4a4"
Nov 03 21:35:10.927591 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:35:10.927547    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:35:13.388261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:13.384762    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:35:13.388261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:13.384934    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z" podUID="b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:14.048909 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:14.048869    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-012641b7-v8vr4"]
Nov 03 21:35:14.095901 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:14.095872    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-012641b7-v8vr4"]
Nov 03 21:35:15.676458 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:15.676417    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e216b0b7-4eef-456e-8f93-3d6c7dac25ad" path="/var/lib/kubelet/pods/e216b0b7-4eef-456e-8f93-3d6c7dac25ad/volumes"
Nov 03 21:35:16.479467 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:16.479425    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4abd935f-ffjcj"]
Nov 03 21:35:16.480111 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:16.480090    5053 cpu_manager.go:414] "containerMap: removing container" podUID="253e3305-deb7-4478-844b-387347d69de0" containerName="revoke-keys"
Nov 03 21:35:16.480225 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:16.480113    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="253e3305-deb7-4478-844b-387347d69de0" containerName="revoke-keys"
Nov 03 21:35:16.480328 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:16.480312    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="253e3305-deb7-4478-844b-387347d69de0" containerName="revoke-keys"
Nov 03 21:35:16.485385 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:16.485364    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abd935f-ffjcj"
Nov 03 21:35:16.541464 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:16.541428    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4abd935f-ffjcj"]
Nov 03 21:35:16.583927 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:16.583893    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:35:16.583927 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:16.583923    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:35:16.584200 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:16.583981    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:35:16.587397 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:16.587372    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:35:16.634196 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:16.634159    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58de2b96-1689-48bb-a906-cf768631fb00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4abd935f-ffjcj\" (UID: \"58de2b96-1689-48bb-a906-cf768631fb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abd935f-ffjcj"
Nov 03 21:35:16.634414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:16.634351    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58de2b96-1689-48bb-a906-cf768631fb00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4abd935f-ffjcj\" (UID: \"58de2b96-1689-48bb-a906-cf768631fb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abd935f-ffjcj"
Nov 03 21:35:16.634524 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:16.634507    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvsgm\" (UniqueName: \"kubernetes.io/projected/58de2b96-1689-48bb-a906-cf768631fb00-kube-api-access-cvsgm\") pod \"maas-api-revoke-sub-4abd935f-ffjcj\" (UID: \"58de2b96-1689-48bb-a906-cf768631fb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abd935f-ffjcj"
Nov 03 21:35:16.736209 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:16.736121    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58de2b96-1689-48bb-a906-cf768631fb00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4abd935f-ffjcj\" (UID: \"58de2b96-1689-48bb-a906-cf768631fb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abd935f-ffjcj"
Nov 03 21:35:16.736209 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:16.736204    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58de2b96-1689-48bb-a906-cf768631fb00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4abd935f-ffjcj\" (UID: \"58de2b96-1689-48bb-a906-cf768631fb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abd935f-ffjcj"
Nov 03 21:35:16.736797 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:16.736350    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvsgm\" (UniqueName: \"kubernetes.io/projected/58de2b96-1689-48bb-a906-cf768631fb00-kube-api-access-cvsgm\") pod \"maas-api-revoke-sub-4abd935f-ffjcj\" (UID: \"58de2b96-1689-48bb-a906-cf768631fb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abd935f-ffjcj"
Nov 03 21:35:16.737029 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:16.737005    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58de2b96-1689-48bb-a906-cf768631fb00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4abd935f-ffjcj\" (UID: \"58de2b96-1689-48bb-a906-cf768631fb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abd935f-ffjcj"
Nov 03 21:35:16.738661 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:16.738642    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58de2b96-1689-48bb-a906-cf768631fb00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4abd935f-ffjcj\" (UID: \"58de2b96-1689-48bb-a906-cf768631fb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abd935f-ffjcj"
Nov 03 21:35:16.793135 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:16.793103    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvsgm\" (UniqueName: \"kubernetes.io/projected/58de2b96-1689-48bb-a906-cf768631fb00-kube-api-access-cvsgm\") pod \"maas-api-revoke-sub-4abd935f-ffjcj\" (UID: \"58de2b96-1689-48bb-a906-cf768631fb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abd935f-ffjcj"
Nov 03 21:35:16.798632 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:16.798607    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abd935f-ffjcj"
Nov 03 21:35:17.187281 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:35:17.187249    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58de2b96_1689_48bb_a906_cf768631fb00.slice/crio-5552a43abc5d5586f522b188126cf2b707d16aa3dce4c60a1c680e6edc1da45a WatchSource:0}: Error finding container 5552a43abc5d5586f522b188126cf2b707d16aa3dce4c60a1c680e6edc1da45a: Status 404 returned error can't find the container with id 5552a43abc5d5586f522b188126cf2b707d16aa3dce4c60a1c680e6edc1da45a
Nov 03 21:35:17.189703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.189676    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4abd935f-ffjcj"]
Nov 03 21:35:17.562603 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.562575    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"]
Nov 03 21:35:17.566813 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.566795    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.669331 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.669303    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c1662df5-openshift-default-dockercfg-g9tfw\""
Nov 03 21:35:17.676696 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.676660    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"]
Nov 03 21:35:17.802122 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.799854    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.802122 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.799899    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.802122 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.800156    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.802122 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.800191    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dzpr\" (UniqueName: \"kubernetes.io/projected/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-kube-api-access-9dzpr\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.802122 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.800298    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.802122 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.800323    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.802122 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.800386    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.802122 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.800425    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.802122 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.800533    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.853333 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.853287    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-182aff01-lfffm"]
Nov 03 21:35:17.876968 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.876434    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-182aff01-lfffm"]
Nov 03 21:35:17.876968 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.876607    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aff01-lfffm"
Nov 03 21:35:17.902317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.902282    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.902564 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.902351    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dzpr\" (UniqueName: \"kubernetes.io/projected/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-kube-api-access-9dzpr\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.902564 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.902426    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.902564 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.902455    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.902564 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.902516    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.902564 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.902547    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.902873 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.902618    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.902873 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.902747    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.902873 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.902770    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.903646 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.903583    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.903811 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.903719    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.904211 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.904106    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.904211 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.904172    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.904641 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.904599    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.905584 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.905559    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.906250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.906218    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.946381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.946341    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abd935f-ffjcj" event={"ID":"58de2b96-1689-48bb-a906-cf768631fb00","Type":"ContainerStarted","Data":"eeb78dcc6af67435cfd48df583a9c1696faf5a4a7ee01b6ae964e64534e6f1cb"}
Nov 03 21:35:17.946576 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.946416    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abd935f-ffjcj" event={"ID":"58de2b96-1689-48bb-a906-cf768631fb00","Type":"ContainerStarted","Data":"5552a43abc5d5586f522b188126cf2b707d16aa3dce4c60a1c680e6edc1da45a"}
Nov 03 21:35:17.960173 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.960135    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dzpr\" (UniqueName: \"kubernetes.io/projected/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-kube-api-access-9dzpr\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:17.960363 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:17.960345    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:18.004172 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:18.004133    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594ecf22-3e9b-4566-a79c-be4dd2b1d85c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-182aff01-lfffm\" (UID: \"594ecf22-3e9b-4566-a79c-be4dd2b1d85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aff01-lfffm"
Nov 03 21:35:18.004407 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:18.004389    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594ecf22-3e9b-4566-a79c-be4dd2b1d85c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-182aff01-lfffm\" (UID: \"594ecf22-3e9b-4566-a79c-be4dd2b1d85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aff01-lfffm"
Nov 03 21:35:18.004497 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:18.004480    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7nqr\" (UniqueName: \"kubernetes.io/projected/594ecf22-3e9b-4566-a79c-be4dd2b1d85c-kube-api-access-w7nqr\") pod \"maas-api-revoke-sub-182aff01-lfffm\" (UID: \"594ecf22-3e9b-4566-a79c-be4dd2b1d85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aff01-lfffm"
Nov 03 21:35:18.105765 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:18.105739    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594ecf22-3e9b-4566-a79c-be4dd2b1d85c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-182aff01-lfffm\" (UID: \"594ecf22-3e9b-4566-a79c-be4dd2b1d85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aff01-lfffm"
Nov 03 21:35:18.105875 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:18.105812    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7nqr\" (UniqueName: \"kubernetes.io/projected/594ecf22-3e9b-4566-a79c-be4dd2b1d85c-kube-api-access-w7nqr\") pod \"maas-api-revoke-sub-182aff01-lfffm\" (UID: \"594ecf22-3e9b-4566-a79c-be4dd2b1d85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aff01-lfffm"
Nov 03 21:35:18.105967 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:18.105953    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594ecf22-3e9b-4566-a79c-be4dd2b1d85c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-182aff01-lfffm\" (UID: \"594ecf22-3e9b-4566-a79c-be4dd2b1d85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aff01-lfffm"
Nov 03 21:35:18.107914 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:18.106761    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594ecf22-3e9b-4566-a79c-be4dd2b1d85c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-182aff01-lfffm\" (UID: \"594ecf22-3e9b-4566-a79c-be4dd2b1d85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aff01-lfffm"
Nov 03 21:35:18.111071 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:18.111045    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594ecf22-3e9b-4566-a79c-be4dd2b1d85c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-182aff01-lfffm\" (UID: \"594ecf22-3e9b-4566-a79c-be4dd2b1d85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aff01-lfffm"
Nov 03 21:35:18.111489 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:18.111451    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abd935f-ffjcj" podStartSLOduration=2.111434839 podStartE2EDuration="2.111434839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:18.107822801 +0000 UTC m=+1232.601744678" watchObservedRunningTime="2026-11-03 21:35:18.111434839 +0000 UTC m=+1232.605356716"
Nov 03 21:35:18.163303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:18.163269    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7nqr\" (UniqueName: \"kubernetes.io/projected/594ecf22-3e9b-4566-a79c-be4dd2b1d85c-kube-api-access-w7nqr\") pod \"maas-api-revoke-sub-182aff01-lfffm\" (UID: \"594ecf22-3e9b-4566-a79c-be4dd2b1d85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aff01-lfffm"
Nov 03 21:35:18.181147 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:18.181114    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:18.196253 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:18.196215    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aff01-lfffm"
Nov 03 21:35:18.609679 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:18.609648    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-182aff01-lfffm"]
Nov 03 21:35:18.612868 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:35:18.612817    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594ecf22_3e9b_4566_a79c_be4dd2b1d85c.slice/crio-757ee0ff44c693019dc118f05bf2e6207c0e764afc58fe86ac58f914e7eb8fa6 WatchSource:0}: Error finding container 757ee0ff44c693019dc118f05bf2e6207c0e764afc58fe86ac58f914e7eb8fa6: Status 404 returned error can't find the container with id 757ee0ff44c693019dc118f05bf2e6207c0e764afc58fe86ac58f914e7eb8fa6
Nov 03 21:35:18.629837 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:18.625423    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"]
Nov 03 21:35:18.639869 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:18.639816    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:35:18.639869 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:18.639915    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:35:18.640295 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:18.639955    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:35:18.958184 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:18.958128    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aff01-lfffm" event={"ID":"594ecf22-3e9b-4566-a79c-be4dd2b1d85c","Type":"ContainerStarted","Data":"757ee0ff44c693019dc118f05bf2e6207c0e764afc58fe86ac58f914e7eb8fa6"}
Nov 03 21:35:18.960368 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:18.960339    5053 generic.go:356] "Generic (PLEG): container finished" podID="58de2b96-1689-48bb-a906-cf768631fb00" containerID="eeb78dcc6af67435cfd48df583a9c1696faf5a4a7ee01b6ae964e64534e6f1cb" exitCode=0
Nov 03 21:35:18.960507 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:18.960422    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abd935f-ffjcj" event={"ID":"58de2b96-1689-48bb-a906-cf768631fb00","Type":"ContainerDied","Data":"eeb78dcc6af67435cfd48df583a9c1696faf5a4a7ee01b6ae964e64534e6f1cb"}
Nov 03 21:35:18.962298 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:18.962265    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb" event={"ID":"9b3caedf-8d81-4f57-b7b5-7d996b5a193a","Type":"ContainerStarted","Data":"098e22c3c107fac956d84a8e7fd019ca523dd73f358b2b276124e5a9b8e609e9"}
Nov 03 21:35:18.962430 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:18.962313    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb" event={"ID":"9b3caedf-8d81-4f57-b7b5-7d996b5a193a","Type":"ContainerStarted","Data":"6eaf40ebe1e8366807a9613251109783732295cdfa03a0732b1335934f36b027"}
Nov 03 21:35:19.182225 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:19.182191    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:19.187566 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:19.187545    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:19.191216 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:19.191173    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb" podStartSLOduration=2.191157757 podStartE2EDuration="2.191157757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:19.186521616 +0000 UTC m=+1233.680443488" watchObservedRunningTime="2026-11-03 21:35:19.191157757 +0000 UTC m=+1233.685079632"
Nov 03 21:35:19.972274 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:19.972227    5053 generic.go:356] "Generic (PLEG): container finished" podID="594ecf22-3e9b-4566-a79c-be4dd2b1d85c" containerID="727a7481af5a37e2b5b58c2fc3957fe512148424d98f07d712425baa1de17d2c" exitCode=0
Nov 03 21:35:19.972274 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:19.972264    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aff01-lfffm" event={"ID":"594ecf22-3e9b-4566-a79c-be4dd2b1d85c","Type":"ContainerDied","Data":"727a7481af5a37e2b5b58c2fc3957fe512148424d98f07d712425baa1de17d2c"}
Nov 03 21:35:19.972802 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:19.972780    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:19.973885 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:19.973869    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:35:20.131205 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:20.131182    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abd935f-ffjcj"
Nov 03 21:35:20.232966 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:20.232878    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58de2b96-1689-48bb-a906-cf768631fb00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58de2b96-1689-48bb-a906-cf768631fb00-maas-api-openshift-ca\") pod \"58de2b96-1689-48bb-a906-cf768631fb00\" (UID: \"58de2b96-1689-48bb-a906-cf768631fb00\") "
Nov 03 21:35:20.232966 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:20.232938    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58de2b96-1689-48bb-a906-cf768631fb00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58de2b96-1689-48bb-a906-cf768631fb00-maas-api-xks-ca\") pod \"58de2b96-1689-48bb-a906-cf768631fb00\" (UID: \"58de2b96-1689-48bb-a906-cf768631fb00\") "
Nov 03 21:35:20.233199 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:20.232995    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58de2b96-1689-48bb-a906-cf768631fb00-kube-api-access-cvsgm\" (UniqueName: \"kubernetes.io/projected/58de2b96-1689-48bb-a906-cf768631fb00-kube-api-access-cvsgm\") pod \"58de2b96-1689-48bb-a906-cf768631fb00\" (UID: \"58de2b96-1689-48bb-a906-cf768631fb00\") "
Nov 03 21:35:20.233507 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:20.233481    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58de2b96-1689-48bb-a906-cf768631fb00-maas-api-openshift-ca" pod "58de2b96-1689-48bb-a906-cf768631fb00" (UID: "58de2b96-1689-48bb-a906-cf768631fb00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:35:20.235852 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:20.234365    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58de2b96-1689-48bb-a906-cf768631fb00-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:35:20.236016 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:20.235992    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58de2b96-1689-48bb-a906-cf768631fb00-maas-api-xks-ca" pod "58de2b96-1689-48bb-a906-cf768631fb00" (UID: "58de2b96-1689-48bb-a906-cf768631fb00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:35:20.237149 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:20.237128    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58de2b96-1689-48bb-a906-cf768631fb00-kube-api-access-cvsgm" pod "58de2b96-1689-48bb-a906-cf768631fb00" (UID: "58de2b96-1689-48bb-a906-cf768631fb00"). InnerVolumeSpecName "kube-api-access-cvsgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:35:20.335718 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:20.335683    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58de2b96-1689-48bb-a906-cf768631fb00-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:35:20.335718 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:20.335712    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvsgm\" (UniqueName: \"kubernetes.io/projected/58de2b96-1689-48bb-a906-cf768631fb00-kube-api-access-cvsgm\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:35:20.982881 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:20.982853    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abd935f-ffjcj"
Nov 03 21:35:20.983303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:20.982881    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abd935f-ffjcj" event={"ID":"58de2b96-1689-48bb-a906-cf768631fb00","Type":"ContainerDied","Data":"5552a43abc5d5586f522b188126cf2b707d16aa3dce4c60a1c680e6edc1da45a"}
Nov 03 21:35:20.983303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:20.982926    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5552a43abc5d5586f522b188126cf2b707d16aa3dce4c60a1c680e6edc1da45a"
Nov 03 21:35:21.150369 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:21.150339    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aff01-lfffm"
Nov 03 21:35:21.248519 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:21.248417    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/594ecf22-3e9b-4566-a79c-be4dd2b1d85c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594ecf22-3e9b-4566-a79c-be4dd2b1d85c-maas-api-openshift-ca\") pod \"594ecf22-3e9b-4566-a79c-be4dd2b1d85c\" (UID: \"594ecf22-3e9b-4566-a79c-be4dd2b1d85c\") "
Nov 03 21:35:21.248519 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:21.248463    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/594ecf22-3e9b-4566-a79c-be4dd2b1d85c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594ecf22-3e9b-4566-a79c-be4dd2b1d85c-maas-api-xks-ca\") pod \"594ecf22-3e9b-4566-a79c-be4dd2b1d85c\" (UID: \"594ecf22-3e9b-4566-a79c-be4dd2b1d85c\") "
Nov 03 21:35:21.248739 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:21.248582    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/594ecf22-3e9b-4566-a79c-be4dd2b1d85c-kube-api-access-w7nqr\" (UniqueName: \"kubernetes.io/projected/594ecf22-3e9b-4566-a79c-be4dd2b1d85c-kube-api-access-w7nqr\") pod \"594ecf22-3e9b-4566-a79c-be4dd2b1d85c\" (UID: \"594ecf22-3e9b-4566-a79c-be4dd2b1d85c\") "
Nov 03 21:35:21.248950 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:21.248834    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/594ecf22-3e9b-4566-a79c-be4dd2b1d85c-maas-api-openshift-ca" pod "594ecf22-3e9b-4566-a79c-be4dd2b1d85c" (UID: "594ecf22-3e9b-4566-a79c-be4dd2b1d85c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:35:21.249286 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:21.249256    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594ecf22-3e9b-4566-a79c-be4dd2b1d85c-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:35:21.251327 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:21.251267    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/594ecf22-3e9b-4566-a79c-be4dd2b1d85c-maas-api-xks-ca" pod "594ecf22-3e9b-4566-a79c-be4dd2b1d85c" (UID: "594ecf22-3e9b-4566-a79c-be4dd2b1d85c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:35:21.251674 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:21.251648    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/594ecf22-3e9b-4566-a79c-be4dd2b1d85c-kube-api-access-w7nqr" pod "594ecf22-3e9b-4566-a79c-be4dd2b1d85c" (UID: "594ecf22-3e9b-4566-a79c-be4dd2b1d85c"). InnerVolumeSpecName "kube-api-access-w7nqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:35:21.350757 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:21.350724    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594ecf22-3e9b-4566-a79c-be4dd2b1d85c-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:35:21.350757 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:21.350750    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7nqr\" (UniqueName: \"kubernetes.io/projected/594ecf22-3e9b-4566-a79c-be4dd2b1d85c-kube-api-access-w7nqr\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:35:21.991514 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:21.991486    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aff01-lfffm"
Nov 03 21:35:21.991981 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:21.991511    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aff01-lfffm" event={"ID":"594ecf22-3e9b-4566-a79c-be4dd2b1d85c","Type":"ContainerDied","Data":"757ee0ff44c693019dc118f05bf2e6207c0e764afc58fe86ac58f914e7eb8fa6"}
Nov 03 21:35:21.991981 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:21.991558    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="757ee0ff44c693019dc118f05bf2e6207c0e764afc58fe86ac58f914e7eb8fa6"
Nov 03 21:35:27.001548 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.001517    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg"]
Nov 03 21:35:27.002280 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.002260    5053 cpu_manager.go:414] "containerMap: removing container" podUID="594ecf22-3e9b-4566-a79c-be4dd2b1d85c" containerName="revoke-keys"
Nov 03 21:35:27.002390 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.002283    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="594ecf22-3e9b-4566-a79c-be4dd2b1d85c" containerName="revoke-keys"
Nov 03 21:35:27.002390 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.002353    5053 cpu_manager.go:414] "containerMap: removing container" podUID="58de2b96-1689-48bb-a906-cf768631fb00" containerName="revoke-keys"
Nov 03 21:35:27.002390 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.002361    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58de2b96-1689-48bb-a906-cf768631fb00" containerName="revoke-keys"
Nov 03 21:35:27.002548 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.002471    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="58de2b96-1689-48bb-a906-cf768631fb00" containerName="revoke-keys"
Nov 03 21:35:27.002548 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.002485    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="594ecf22-3e9b-4566-a79c-be4dd2b1d85c" containerName="revoke-keys"
Nov 03 21:35:27.007007 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.006985    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg"
Nov 03 21:35:27.063102 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.063066    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg"]
Nov 03 21:35:27.122712 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.109626    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c1662df5-dockercfg-96xww\""
Nov 03 21:35:27.123114 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.123091    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c1662df5\""
Nov 03 21:35:27.136258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.132558    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7hwf\" (UniqueName: \"kubernetes.io/projected/5553e95b-04c9-4716-a5c6-8f5cdf96bbd2-kube-api-access-f7hwf\") pod \"payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg\" (UID: \"5553e95b-04c9-4716-a5c6-8f5cdf96bbd2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg"
Nov 03 21:35:27.136258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.132694    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5553e95b-04c9-4716-a5c6-8f5cdf96bbd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg\" (UID: \"5553e95b-04c9-4716-a5c6-8f5cdf96bbd2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg"
Nov 03 21:35:27.229275 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.228198    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm"]
Nov 03 21:35:27.235943 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.234609    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7hwf\" (UniqueName: \"kubernetes.io/projected/5553e95b-04c9-4716-a5c6-8f5cdf96bbd2-kube-api-access-f7hwf\") pod \"payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg\" (UID: \"5553e95b-04c9-4716-a5c6-8f5cdf96bbd2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg"
Nov 03 21:35:27.235943 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.234746    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5553e95b-04c9-4716-a5c6-8f5cdf96bbd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg\" (UID: \"5553e95b-04c9-4716-a5c6-8f5cdf96bbd2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg"
Nov 03 21:35:27.235943 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.235657    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5553e95b-04c9-4716-a5c6-8f5cdf96bbd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg\" (UID: \"5553e95b-04c9-4716-a5c6-8f5cdf96bbd2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg"
Nov 03 21:35:27.239057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.239025    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm"
Nov 03 21:35:27.256894 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.256826    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm"]
Nov 03 21:35:27.292062 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.292032    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7hwf\" (UniqueName: \"kubernetes.io/projected/5553e95b-04c9-4716-a5c6-8f5cdf96bbd2-kube-api-access-f7hwf\") pod \"payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg\" (UID: \"5553e95b-04c9-4716-a5c6-8f5cdf96bbd2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg"
Nov 03 21:35:27.324813 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.324783    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg"
Nov 03 21:35:27.336092 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.336058    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ead33cd-8e19-422f-bef3-4e069ba97eda-plugins-config-volume\") pod \"payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm\" (UID: \"0ead33cd-8e19-422f-bef3-4e069ba97eda\") " pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm"
Nov 03 21:35:27.336300 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.336269    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84tpq\" (UniqueName: \"kubernetes.io/projected/0ead33cd-8e19-422f-bef3-4e069ba97eda-kube-api-access-84tpq\") pod \"payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm\" (UID: \"0ead33cd-8e19-422f-bef3-4e069ba97eda\") " pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm"
Nov 03 21:35:27.438202 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.438158    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ead33cd-8e19-422f-bef3-4e069ba97eda-plugins-config-volume\") pod \"payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm\" (UID: \"0ead33cd-8e19-422f-bef3-4e069ba97eda\") " pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm"
Nov 03 21:35:27.438398 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.438334    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84tpq\" (UniqueName: \"kubernetes.io/projected/0ead33cd-8e19-422f-bef3-4e069ba97eda-kube-api-access-84tpq\") pod \"payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm\" (UID: \"0ead33cd-8e19-422f-bef3-4e069ba97eda\") " pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm"
Nov 03 21:35:27.440340 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.439909    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ead33cd-8e19-422f-bef3-4e069ba97eda-plugins-config-volume\") pod \"payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm\" (UID: \"0ead33cd-8e19-422f-bef3-4e069ba97eda\") " pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm"
Nov 03 21:35:27.498116 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.498089    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84tpq\" (UniqueName: \"kubernetes.io/projected/0ead33cd-8e19-422f-bef3-4e069ba97eda-kube-api-access-84tpq\") pod \"payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm\" (UID: \"0ead33cd-8e19-422f-bef3-4e069ba97eda\") " pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm"
Nov 03 21:35:27.557712 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.557621    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm"
Nov 03 21:35:27.709110 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.709080    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg"]
Nov 03 21:35:27.948515 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:35:27.948483    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ead33cd_8e19_422f_bef3_4e069ba97eda.slice/crio-a9ae6a5c2aaf25cd5f3ec95f02f582d35cd14ddc0b88a9990dfea60d3f1d05e3 WatchSource:0}: Error finding container a9ae6a5c2aaf25cd5f3ec95f02f582d35cd14ddc0b88a9990dfea60d3f1d05e3: Status 404 returned error can't find the container with id a9ae6a5c2aaf25cd5f3ec95f02f582d35cd14ddc0b88a9990dfea60d3f1d05e3
Nov 03 21:35:27.951944 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:27.951922    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm"]
Nov 03 21:35:28.044616 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:28.044583    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm" event={"ID":"0ead33cd-8e19-422f-bef3-4e069ba97eda","Type":"ContainerStarted","Data":"f8254eebaa7afc825f982670652adaf8ad2e403db598a995ed376a541ba30427"}
Nov 03 21:35:28.045081 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:28.044647    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm" event={"ID":"0ead33cd-8e19-422f-bef3-4e069ba97eda","Type":"ContainerStarted","Data":"a9ae6a5c2aaf25cd5f3ec95f02f582d35cd14ddc0b88a9990dfea60d3f1d05e3"}
Nov 03 21:35:28.045081 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:28.044673    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm"
Nov 03 21:35:28.046624 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:28.046597    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg" event={"ID":"5553e95b-04c9-4716-a5c6-8f5cdf96bbd2","Type":"ContainerStarted","Data":"b77c5e59ae830c46d2a13719615f34c92196610eb4e1700253ec7f85a9b5f26f"}
Nov 03 21:35:28.046770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:28.046653    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg"
Nov 03 21:35:28.046770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:28.046676    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg" event={"ID":"5553e95b-04c9-4716-a5c6-8f5cdf96bbd2","Type":"ContainerStarted","Data":"43d7ed6b6bb4b48c9a64f3c27f57dcff220e1d8d94e44643b679455243122029"}
Nov 03 21:35:28.159974 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:28.159914    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm" podStartSLOduration=1.159901137 podStartE2EDuration="1.159901137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:28.157573436 +0000 UTC m=+1242.651495312" watchObservedRunningTime="2026-11-03 21:35:28.159901137 +0000 UTC m=+1242.653823011"
Nov 03 21:35:28.272508 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:28.272260    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg" podStartSLOduration=2.2722232079999998 podStartE2EDuration="2.272223208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:28.27114137 +0000 UTC m=+1242.765063247" watchObservedRunningTime="2026-11-03 21:35:28.272223208 +0000 UTC m=+1242.766145084"
Nov 03 21:35:30.084546 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:30.084513    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6057588-b6cjm"]
Nov 03 21:35:30.122806 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:30.122772    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6057588-b6cjm"]
Nov 03 21:35:31.632255 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:31.632107    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx" podUID="a6c0d3aa-18c2-465b-846b-b82968acc780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:31.712263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:31.711506    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc752551-d8bd-439b-8cae-eb0faaf1ecb2" path="/var/lib/kubelet/pods/fc752551-d8bd-439b-8cae-eb0faaf1ecb2/volumes"
Nov 03 21:35:31.716258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:31.713483    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n" podUID="512b3ab1-45ba-4f96-9799-b129b25fca42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:31.936325 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:31.932130    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8" podUID="43404f1c-c87c-4325-ab97-bffb6ea81e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:32.366263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:32.366038    5053 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n" podUID="512b3ab1-45ba-4f96-9799-b129b25fca42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:32.576261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:32.575365    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd" podUID="82077ea7-1e05-4b6e-806b-a2d2d2a64c4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:32.602262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:32.591554    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6" podUID="4803807f-9312-40ac-badb-00c00d3f3bd3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:32.636322 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:32.635264    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx" podUID="a6c0d3aa-18c2-465b-846b-b82968acc780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:32.636322 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:32.635674    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:35:32.715679 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:32.714005    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n" podUID="512b3ab1-45ba-4f96-9799-b129b25fca42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:32.715679 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:32.714412    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:35:32.867259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:32.866360    5053 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd" podUID="82077ea7-1e05-4b6e-806b-a2d2d2a64c4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:32.963388 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:32.963320    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8" podUID="43404f1c-c87c-4325-ab97-bffb6ea81e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:32.968674 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:32.968637    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:35:33.578567 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:33.578502    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd" podUID="82077ea7-1e05-4b6e-806b-a2d2d2a64c4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:33.595291 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:33.583423    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:35:33.619334 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:33.595647    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6" podUID="4803807f-9312-40ac-badb-00c00d3f3bd3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:33.642118 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:33.639739    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx" podUID="a6c0d3aa-18c2-465b-846b-b82968acc780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 03 21:35:33.642118 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:33.641273    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:35:33.717018 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:33.716974    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n" podUID="512b3ab1-45ba-4f96-9799-b129b25fca42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:33.851661 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:33.851578    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 21:35:33.877284 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:33.864087    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:35:34.241462 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:34.241428    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 21:35:34.273385 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:34.273340    5053 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx" podUID="a6c0d3aa-18c2-465b-846b-b82968acc780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded"
Nov 03 21:35:34.276421 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:34.276395    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 21:35:34.289164 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:34.289114    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 21:35:34.321963 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:34.321935    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62a8bad8-dcxvw"]
Nov 03 21:35:34.454315 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:34.454279    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62a8bad8-dcxvw"]
Nov 03 21:35:34.825331 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:34.825285    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"]
Nov 03 21:35:34.848039 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:34.848010    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:34.862413 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:34.862384    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"]
Nov 03 21:35:34.985105 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:34.984264    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-envoy\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:34.985654 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:34.985628    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx7q2\" (UniqueName: \"kubernetes.io/projected/18feb2eb-b1a3-492c-8699-e507bf82b55f-kube-api-access-hx7q2\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:34.985888 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:34.985867    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-workload-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:34.986035 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:34.986018    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-credential-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:34.986201 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:34.986182    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-podinfo\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:34.986524 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:34.986503    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18feb2eb-b1a3-492c-8699-e507bf82b55f-istiod-ca-cert\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:34.986721 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:34.986702    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-workload-certs\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:34.986905 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:34.986889    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-data\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:34.987082 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:34.987066    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-token\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:35.097718 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.097622    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-token\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:35.097995 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.097972    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-envoy\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:35.098184 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.098169    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx7q2\" (UniqueName: \"kubernetes.io/projected/18feb2eb-b1a3-492c-8699-e507bf82b55f-kube-api-access-hx7q2\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:35.098373 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.098353    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-workload-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:35.098489 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.098474    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-credential-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:35.098662 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.098646    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-podinfo\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:35.098823 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.098809    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18feb2eb-b1a3-492c-8699-e507bf82b55f-istiod-ca-cert\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:35.098967 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.098952    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-workload-certs\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:35.099085 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.099069    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-data\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:35.099655 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.099632    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-data\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:35.104255 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.100921    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-credential-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:35.105063 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.105040    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-envoy\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:35.138542 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.136144    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-workload-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:35.142385 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.140587    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-workload-certs\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:35.142385 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.141134    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-podinfo\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:35.143111 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.143087    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18feb2eb-b1a3-492c-8699-e507bf82b55f-istiod-ca-cert\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:35.183084 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.182635    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx7q2\" (UniqueName: \"kubernetes.io/projected/18feb2eb-b1a3-492c-8699-e507bf82b55f-kube-api-access-hx7q2\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:35.187471 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.187434    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-token\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:35.192271 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.192227    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:35.672377 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.672341    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="827b72b2-5ec8-4186-9cba-e2c639fedde9" path="/var/lib/kubelet/pods/827b72b2-5ec8-4186-9cba-e2c639fedde9/volumes"
Nov 03 21:35:35.682450 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.682427    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"]
Nov 03 21:35:35.684075 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:35:35.684050    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18feb2eb_b1a3_492c_8699_e507bf82b55f.slice/crio-5ee9cce5ff1731a7645851b50aaf22d77748bb5d71f6a95b857f5b8e8b3d68d1 WatchSource:0}: Error finding container 5ee9cce5ff1731a7645851b50aaf22d77748bb5d71f6a95b857f5b8e8b3d68d1: Status 404 returned error can't find the container with id 5ee9cce5ff1731a7645851b50aaf22d77748bb5d71f6a95b857f5b8e8b3d68d1
Nov 03 21:35:35.686931 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.686898    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:35:35.687034 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.686973    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:35:35.687034 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.687015    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:35:35.717858 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.717824    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh" event={"ID":"18feb2eb-b1a3-492c-8699-e507bf82b55f","Type":"ContainerStarted","Data":"5ee9cce5ff1731a7645851b50aaf22d77748bb5d71f6a95b857f5b8e8b3d68d1"}
Nov 03 21:35:35.746990 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.746962    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c325017c-qkr55"]
Nov 03 21:35:35.752639 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.752617    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325017c-qkr55"
Nov 03 21:35:35.805700 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.804880    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:35:35.805700 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.805006    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:35:35.807438 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.807403    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c325017c-qkr55"]
Nov 03 21:35:35.811012 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.810590    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d606955a-bcbf-44cc-9d76-7601ad0b3e94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c325017c-qkr55\" (UID: \"d606955a-bcbf-44cc-9d76-7601ad0b3e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325017c-qkr55"
Nov 03 21:35:35.811012 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.810760    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2tfj\" (UniqueName: \"kubernetes.io/projected/d606955a-bcbf-44cc-9d76-7601ad0b3e94-kube-api-access-k2tfj\") pod \"maas-api-revoke-sub-c325017c-qkr55\" (UID: \"d606955a-bcbf-44cc-9d76-7601ad0b3e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325017c-qkr55"
Nov 03 21:35:35.811012 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.810801    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d606955a-bcbf-44cc-9d76-7601ad0b3e94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c325017c-qkr55\" (UID: \"d606955a-bcbf-44cc-9d76-7601ad0b3e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325017c-qkr55"
Nov 03 21:35:35.884340 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.884313    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:35:35.884340 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.884314    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:35:35.911924 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.911897    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d606955a-bcbf-44cc-9d76-7601ad0b3e94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c325017c-qkr55\" (UID: \"d606955a-bcbf-44cc-9d76-7601ad0b3e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325017c-qkr55"
Nov 03 21:35:35.912061 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.911979    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2tfj\" (UniqueName: \"kubernetes.io/projected/d606955a-bcbf-44cc-9d76-7601ad0b3e94-kube-api-access-k2tfj\") pod \"maas-api-revoke-sub-c325017c-qkr55\" (UID: \"d606955a-bcbf-44cc-9d76-7601ad0b3e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325017c-qkr55"
Nov 03 21:35:35.912061 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.911997    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d606955a-bcbf-44cc-9d76-7601ad0b3e94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c325017c-qkr55\" (UID: \"d606955a-bcbf-44cc-9d76-7601ad0b3e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325017c-qkr55"
Nov 03 21:35:35.912589 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.912568    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d606955a-bcbf-44cc-9d76-7601ad0b3e94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c325017c-qkr55\" (UID: \"d606955a-bcbf-44cc-9d76-7601ad0b3e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325017c-qkr55"
Nov 03 21:35:35.914494 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.914473    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d606955a-bcbf-44cc-9d76-7601ad0b3e94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c325017c-qkr55\" (UID: \"d606955a-bcbf-44cc-9d76-7601ad0b3e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325017c-qkr55"
Nov 03 21:35:35.969041 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:35.969009    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2tfj\" (UniqueName: \"kubernetes.io/projected/d606955a-bcbf-44cc-9d76-7601ad0b3e94-kube-api-access-k2tfj\") pod \"maas-api-revoke-sub-c325017c-qkr55\" (UID: \"d606955a-bcbf-44cc-9d76-7601ad0b3e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325017c-qkr55"
Nov 03 21:35:36.072776 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:36.072740    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325017c-qkr55"
Nov 03 21:35:36.467081 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:36.467050    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c325017c-qkr55"]
Nov 03 21:35:36.467196 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:35:36.467131    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd606955a_bcbf_44cc_9d76_7601ad0b3e94.slice/crio-1220d918d383e8f3e28a845141cb3d8b2ccd02dcbe935ce3cb33ec2f66ed5f89 WatchSource:0}: Error finding container 1220d918d383e8f3e28a845141cb3d8b2ccd02dcbe935ce3cb33ec2f66ed5f89: Status 404 returned error can't find the container with id 1220d918d383e8f3e28a845141cb3d8b2ccd02dcbe935ce3cb33ec2f66ed5f89
Nov 03 21:35:36.734874 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:36.734783    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh" event={"ID":"18feb2eb-b1a3-492c-8699-e507bf82b55f","Type":"ContainerStarted","Data":"40aac3ccec408709b293c389e67206c2382e7b8ab290222277764b399f8b8669"}
Nov 03 21:35:36.736147 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:36.736124    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325017c-qkr55" event={"ID":"d606955a-bcbf-44cc-9d76-7601ad0b3e94","Type":"ContainerStarted","Data":"1220d918d383e8f3e28a845141cb3d8b2ccd02dcbe935ce3cb33ec2f66ed5f89"}
Nov 03 21:35:36.855675 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:36.855612    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh" podStartSLOduration=2.85559384 podStartE2EDuration="2.85559384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:36.850569184 +0000 UTC m=+1251.344491083" watchObservedRunningTime="2026-11-03 21:35:36.85559384 +0000 UTC m=+1251.349515715"
Nov 03 21:35:37.193147 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:37.193116    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:37.194926 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:37.194905    5053 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 03 21:35:37.194982 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:37.194959    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh" podUID="18feb2eb-b1a3-492c-8699-e507bf82b55f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 03 21:35:37.748706 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:37.748668    5053 generic.go:356] "Generic (PLEG): container finished" podID="d606955a-bcbf-44cc-9d76-7601ad0b3e94" containerID="783333e23a74ba617043005d53804302f111ed3e2e0d24d9093fd83383d753a1" exitCode=0
Nov 03 21:35:37.748706 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:37.748691    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325017c-qkr55" event={"ID":"d606955a-bcbf-44cc-9d76-7601ad0b3e94","Type":"ContainerDied","Data":"783333e23a74ba617043005d53804302f111ed3e2e0d24d9093fd83383d753a1"}
Nov 03 21:35:38.193406 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:38.193373    5053 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 03 21:35:38.193803 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:38.193429    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh" podUID="18feb2eb-b1a3-492c-8699-e507bf82b55f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 03 21:35:38.964558 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:38.964532    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325017c-qkr55"
Nov 03 21:35:39.062174 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:39.062100    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg"
Nov 03 21:35:39.062174 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:39.062166    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm"
Nov 03 21:35:39.066874 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:39.066854    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d606955a-bcbf-44cc-9d76-7601ad0b3e94-kube-api-access-k2tfj\" (UniqueName: \"kubernetes.io/projected/d606955a-bcbf-44cc-9d76-7601ad0b3e94-kube-api-access-k2tfj\") pod \"d606955a-bcbf-44cc-9d76-7601ad0b3e94\" (UID: \"d606955a-bcbf-44cc-9d76-7601ad0b3e94\") "
Nov 03 21:35:39.066948 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:39.066927    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d606955a-bcbf-44cc-9d76-7601ad0b3e94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d606955a-bcbf-44cc-9d76-7601ad0b3e94-maas-api-xks-ca\") pod \"d606955a-bcbf-44cc-9d76-7601ad0b3e94\" (UID: \"d606955a-bcbf-44cc-9d76-7601ad0b3e94\") "
Nov 03 21:35:39.067082 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:39.067069    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d606955a-bcbf-44cc-9d76-7601ad0b3e94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d606955a-bcbf-44cc-9d76-7601ad0b3e94-maas-api-openshift-ca\") pod \"d606955a-bcbf-44cc-9d76-7601ad0b3e94\" (UID: \"d606955a-bcbf-44cc-9d76-7601ad0b3e94\") "
Nov 03 21:35:39.067405 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:39.067379    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d606955a-bcbf-44cc-9d76-7601ad0b3e94-maas-api-openshift-ca" pod "d606955a-bcbf-44cc-9d76-7601ad0b3e94" (UID: "d606955a-bcbf-44cc-9d76-7601ad0b3e94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:35:39.069114 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:39.069089    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d606955a-bcbf-44cc-9d76-7601ad0b3e94-maas-api-xks-ca" pod "d606955a-bcbf-44cc-9d76-7601ad0b3e94" (UID: "d606955a-bcbf-44cc-9d76-7601ad0b3e94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:35:39.069194 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:39.069112    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d606955a-bcbf-44cc-9d76-7601ad0b3e94-kube-api-access-k2tfj" pod "d606955a-bcbf-44cc-9d76-7601ad0b3e94" (UID: "d606955a-bcbf-44cc-9d76-7601ad0b3e94"). InnerVolumeSpecName "kube-api-access-k2tfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:35:39.168229 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:39.168193    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d606955a-bcbf-44cc-9d76-7601ad0b3e94-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:35:39.168229 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:39.168221    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2tfj\" (UniqueName: \"kubernetes.io/projected/d606955a-bcbf-44cc-9d76-7601ad0b3e94-kube-api-access-k2tfj\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:35:39.168520 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:39.168232    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d606955a-bcbf-44cc-9d76-7601ad0b3e94-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:35:39.193042 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:39.193010    5053 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 03 21:35:39.193186 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:39.193065    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh" podUID="18feb2eb-b1a3-492c-8699-e507bf82b55f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 03 21:35:39.765347 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:39.765315    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325017c-qkr55" event={"ID":"d606955a-bcbf-44cc-9d76-7601ad0b3e94","Type":"ContainerDied","Data":"1220d918d383e8f3e28a845141cb3d8b2ccd02dcbe935ce3cb33ec2f66ed5f89"}
Nov 03 21:35:39.765722 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:39.765352    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325017c-qkr55"
Nov 03 21:35:39.765722 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:39.765369    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1220d918d383e8f3e28a845141cb3d8b2ccd02dcbe935ce3cb33ec2f66ed5f89"
Nov 03 21:35:40.192771 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:40.192717    5053 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 03 21:35:40.192978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:40.192789    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh" podUID="18feb2eb-b1a3-492c-8699-e507bf82b55f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 03 21:35:40.955684 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:35:40.955646    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:35:41.192695 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:41.192658    5053 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 03 21:35:41.192856 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:41.192708    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh" podUID="18feb2eb-b1a3-492c-8699-e507bf82b55f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 03 21:35:42.193594 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:42.193551    5053 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 03 21:35:42.194112 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:42.193616    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh" podUID="18feb2eb-b1a3-492c-8699-e507bf82b55f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 03 21:35:43.197618 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:43.197585    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:43.198092 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:43.198070    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:43.198921 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:43.198904    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:35:45.047911 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:45.047875    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08d344e-q69kq"]
Nov 03 21:35:45.100675 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:45.100635    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08d344e-q69kq"]
Nov 03 21:35:45.720535 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:45.720497    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e467c2a-8216-45d3-b354-8c86d0931091" path="/var/lib/kubelet/pods/5e467c2a-8216-45d3-b354-8c86d0931091/volumes"
Nov 03 21:35:46.073709 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:46.073616    5053 scope.go:122] "RemoveContainer" containerID="24fcbd5704f938123df1900914a29a4cfa879954cd9e6d2729c4dcbb60fae7d5"
Nov 03 21:35:53.048842 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:53.048805    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51eeb46f-qr5gs"]
Nov 03 21:35:53.098307 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:53.098265    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51eeb46f-qr5gs"]
Nov 03 21:35:53.668854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:53.668823    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99a9cf09-1a48-4a77-9127-9ceb9279708e" path="/var/lib/kubelet/pods/99a9cf09-1a48-4a77-9127-9ceb9279708e/volumes"
Nov 03 21:35:57.579562 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.579527    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"]
Nov 03 21:35:57.580203 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.580189    5053 cpu_manager.go:414] "containerMap: removing container" podUID="d606955a-bcbf-44cc-9d76-7601ad0b3e94" containerName="revoke-keys"
Nov 03 21:35:57.580291 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.580204    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d606955a-bcbf-44cc-9d76-7601ad0b3e94" containerName="revoke-keys"
Nov 03 21:35:57.580365 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.580355    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="d606955a-bcbf-44cc-9d76-7601ad0b3e94" containerName="revoke-keys"
Nov 03 21:35:57.585766 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.585748    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.693893 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.693860    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.694111 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.693935    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/190b08f2-f879-4a61-aee1-590afc3c4afc-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.694111 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.693997    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.694111 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.694021    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.694111 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.694047    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.694111 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.694063    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbcwh\" (UniqueName: \"kubernetes.io/projected/190b08f2-f879-4a61-aee1-590afc3c4afc-kube-api-access-zbcwh\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.694416 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.694216    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.694416 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.694330    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.694416 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.694403    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.694612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.694421    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"]
Nov 03 21:35:57.795999 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.795950    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/190b08f2-f879-4a61-aee1-590afc3c4afc-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.795999 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.796002    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.796426 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.796034    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.796426 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.796069    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.796426 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.796091    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbcwh\" (UniqueName: \"kubernetes.io/projected/190b08f2-f879-4a61-aee1-590afc3c4afc-kube-api-access-zbcwh\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.796426 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.796140    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.796426 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.796218    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.796426 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.796330    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.796715 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.796478    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.796715 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.796687    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.796834 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.796723    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.796834 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.796787    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.796914 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.796843    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/190b08f2-f879-4a61-aee1-590afc3c4afc-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.796962 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.796943    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.798647 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.798622    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.798911 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.798893    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.854900 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.854833    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbcwh\" (UniqueName: \"kubernetes.io/projected/190b08f2-f879-4a61-aee1-590afc3c4afc-kube-api-access-zbcwh\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.855394 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.855373    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:57.899727 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:57.899685    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:58.331559 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:35:58.331511    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190b08f2_f879_4a61_aee1_590afc3c4afc.slice/crio-cae4c5b8a01232f190a02c9060d9293e78a6c146f01721518f2f859362faf054 WatchSource:0}: Error finding container cae4c5b8a01232f190a02c9060d9293e78a6c146f01721518f2f859362faf054: Status 404 returned error can't find the container with id cae4c5b8a01232f190a02c9060d9293e78a6c146f01721518f2f859362faf054
Nov 03 21:35:58.332698 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:58.332671    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"]
Nov 03 21:35:58.334497 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:58.334466    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:35:58.334595 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:58.334541    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:35:58.334595 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:58.334588    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:35:59.031390 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:59.018083    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g" event={"ID":"190b08f2-f879-4a61-aee1-590afc3c4afc","Type":"ContainerStarted","Data":"b8ca2e535ba553875a6ed12efc532eda12d1a481def097fc7226e626b9517c7b"}
Nov 03 21:35:59.031390 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:59.018152    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g" event={"ID":"190b08f2-f879-4a61-aee1-590afc3c4afc","Type":"ContainerStarted","Data":"cae4c5b8a01232f190a02c9060d9293e78a6c146f01721518f2f859362faf054"}
Nov 03 21:35:59.139452 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:59.139390    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g" podStartSLOduration=2.139371742 podStartE2EDuration="2.139371742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:59.133882346 +0000 UTC m=+1273.627804222" watchObservedRunningTime="2026-11-03 21:35:59.139371742 +0000 UTC m=+1273.633293616"
Nov 03 21:35:59.900343 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:59.900309    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:35:59.907259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:59.902093    5053 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 03 21:35:59.907259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:35:59.902147    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g" podUID="190b08f2-f879-4a61-aee1-590afc3c4afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 03 21:36:00.924262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:00.920698    5053 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 03 21:36:00.924262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:00.920765    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g" podUID="190b08f2-f879-4a61-aee1-590afc3c4afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 03 21:36:01.911265 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:01.909673    5053 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 03 21:36:01.911265 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:01.909762    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g" podUID="190b08f2-f879-4a61-aee1-590afc3c4afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 03 21:36:02.904260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:02.901674    5053 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 03 21:36:02.904260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:02.901753    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g" podUID="190b08f2-f879-4a61-aee1-590afc3c4afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 03 21:36:03.900129 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:03.900099    5053 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 03 21:36:03.900389 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:03.900151    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g" podUID="190b08f2-f879-4a61-aee1-590afc3c4afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 03 21:36:04.883735 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:04.883690    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"]
Nov 03 21:36:04.889045 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:04.889025    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:04.900513 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:04.900487    5053 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 03 21:36:04.900722 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:04.900693    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g" podUID="190b08f2-f879-4a61-aee1-590afc3c4afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 03 21:36:04.998000 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:04.997968    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"]
Nov 03 21:36:05.010502 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.010453    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-workload-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.010675 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.010503    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-credential-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.010675 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.010601    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-token\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.010675 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.010627    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn6gx\" (UniqueName: \"kubernetes.io/projected/1743870f-2488-4b26-a01d-baa5b81ca42d-kube-api-access-qn6gx\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.010675 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.010666    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-envoy\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.010897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.010746    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-workload-certs\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.010897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.010803    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-data\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.010897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.010829    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-podinfo\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.010897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.010863    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1743870f-2488-4b26-a01d-baa5b81ca42d-istiod-ca-cert\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.112181 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.112134    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1743870f-2488-4b26-a01d-baa5b81ca42d-istiod-ca-cert\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.112404 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.112349    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-workload-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.112404 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.112375    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-credential-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.112521 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.112456    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-token\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.112521 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.112482    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn6gx\" (UniqueName: \"kubernetes.io/projected/1743870f-2488-4b26-a01d-baa5b81ca42d-kube-api-access-qn6gx\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.112627 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.112518    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-envoy\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.112627 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.112590    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-workload-certs\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.112790 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.112639    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-data\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.112790 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.112662    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-podinfo\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.113475 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.113444    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-workload-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.113608 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.113486    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-credential-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.114038 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.114010    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-workload-certs\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.114179 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.114152    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1743870f-2488-4b26-a01d-baa5b81ca42d-istiod-ca-cert\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.114323 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.114295    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-data\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.115903 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.115879    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-podinfo\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.116192 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.116168    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-envoy\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.169627 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.169562    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-token\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.169901 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.169877    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn6gx\" (UniqueName: \"kubernetes.io/projected/1743870f-2488-4b26-a01d-baa5b81ca42d-kube-api-access-qn6gx\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.215125 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.215079    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:05.665108 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.665070    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:36:05.665263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.665151    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:36:05.665263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.665192    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:36:05.671506 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.671481    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"]
Nov 03 21:36:05.905687 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.905655    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:36:05.906163 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.906072    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:36:05.906879 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:05.906852    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:36:06.084738 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:06.084704    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm" event={"ID":"1743870f-2488-4b26-a01d-baa5b81ca42d","Type":"ContainerStarted","Data":"9fb0e93f07d7f62092ee7445bb273e3abe61635581d15cfd2e129e4b4da49692"}
Nov 03 21:36:06.084904 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:06.084759    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm" event={"ID":"1743870f-2488-4b26-a01d-baa5b81ca42d","Type":"ContainerStarted","Data":"3b43f8ddd4fa7586b5d966dc9e3442d1ef2881b08e5900112b9ae5857af117e4"}
Nov 03 21:36:06.215972 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:06.215900    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:06.217503 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:06.217474    5053 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 03 21:36:06.217618 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:06.217522    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm" podUID="1743870f-2488-4b26-a01d-baa5b81ca42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 03 21:36:06.265178 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:06.265117    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm" podStartSLOduration=2.265096177 podStartE2EDuration="2.265096177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:36:06.258598482 +0000 UTC m=+1280.752520358" watchObservedRunningTime="2026-11-03 21:36:06.265096177 +0000 UTC m=+1280.759018053"
Nov 03 21:36:07.221261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:07.220665    5053 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 03 21:36:07.221261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:07.220724    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm" podUID="1743870f-2488-4b26-a01d-baa5b81ca42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 03 21:36:08.220735 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:08.220697    5053 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 03 21:36:08.221053 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:08.221009    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm" podUID="1743870f-2488-4b26-a01d-baa5b81ca42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 03 21:36:09.222786 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:09.222747    5053 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 03 21:36:09.223432 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:09.223384    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm" podUID="1743870f-2488-4b26-a01d-baa5b81ca42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 03 21:36:10.241267 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:10.240399    5053 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 03 21:36:10.241267 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:10.240478    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm" podUID="1743870f-2488-4b26-a01d-baa5b81ca42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 03 21:36:11.153604 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:36:11.153547    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:36:11.227261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:11.227062    5053 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 03 21:36:11.227261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:11.227123    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm" podUID="1743870f-2488-4b26-a01d-baa5b81ca42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 03 21:36:11.643845 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:11.642664    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd" podUID="82077ea7-1e05-4b6e-806b-a2d2d2a64c4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:36:12.216826 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:12.216787    5053 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 03 21:36:12.217177 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:12.217124    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm" podUID="1743870f-2488-4b26-a01d-baa5b81ca42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 03 21:36:12.253396 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:12.253351    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6" podUID="4803807f-9312-40ac-badb-00c00d3f3bd3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 21:36:12.269356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:12.269318    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-blk95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:36:12.269573 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:12.269380    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95" podUID="48c6d17e-4e96-4837-8ec2-c1da7ecf30be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:36:12.400267 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:12.399412    5053 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n" podUID="512b3ab1-45ba-4f96-9799-b129b25fca42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:36:12.657337 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:12.646912    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd" podUID="82077ea7-1e05-4b6e-806b-a2d2d2a64c4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:36:12.679664 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:12.679635    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:36:12.737279 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:12.724768    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n" podUID="512b3ab1-45ba-4f96-9799-b129b25fca42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:36:12.849912 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:12.849866    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8" podUID="43404f1c-c87c-4325-ab97-bffb6ea81e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Nov 03 21:36:12.871355 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:12.852024    5053 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd" podUID="82077ea7-1e05-4b6e-806b-a2d2d2a64c4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:36:12.966326 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:12.942866    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 21:36:13.216815 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:13.216713    5053 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 03 21:36:13.236262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:13.217042    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm" podUID="1743870f-2488-4b26-a01d-baa5b81ca42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 03 21:36:14.331611 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:14.331577    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:14.332906 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:14.332880    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:14.342498 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:14.342465    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:36:19.992384 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:19.991788    5053 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": read tcp 10.132.0.2:56838->10.132.0.126:15021: read: connection reset by peer" start-of-body=
Nov 03 21:36:19.992384 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:19.991867    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb" podUID="9b3caedf-8d81-4f57-b7b5-7d996b5a193a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": read tcp 10.132.0.2:56838->10.132.0.126:15021: read: connection reset by peer"
Nov 03 21:36:20.401167 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:20.401074    5053 generic.go:356] "Generic (PLEG): container finished" podID="9b3caedf-8d81-4f57-b7b5-7d996b5a193a" containerID="098e22c3c107fac956d84a8e7fd019ca523dd73f358b2b276124e5a9b8e609e9" exitCode=0
Nov 03 21:36:20.401167 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:20.401146    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb" event={"ID":"9b3caedf-8d81-4f57-b7b5-7d996b5a193a","Type":"ContainerDied","Data":"098e22c3c107fac956d84a8e7fd019ca523dd73f358b2b276124e5a9b8e609e9"}
Nov 03 21:36:20.401634 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:20.401619    5053 scope.go:122] "RemoveContainer" containerID="098e22c3c107fac956d84a8e7fd019ca523dd73f358b2b276124e5a9b8e609e9"
Nov 03 21:36:20.403118 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:20.403093    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:36:20.403186 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:20.403158    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:36:20.403217 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:20.403189    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:36:21.431612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:21.431575    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb" event={"ID":"9b3caedf-8d81-4f57-b7b5-7d996b5a193a","Type":"ContainerStarted","Data":"929030f1cdeda8ff6b7c9210503a446de153892e3a81580f4046d10b40c9fce4"}
Nov 03 21:36:22.182018 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:22.181982    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:36:22.182225 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:22.182206    5053 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 03 21:36:22.182306 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:22.182282    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb" podUID="9b3caedf-8d81-4f57-b7b5-7d996b5a193a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 03 21:36:23.182128 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:23.182092    5053 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 03 21:36:23.182723 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:23.182172    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb" podUID="9b3caedf-8d81-4f57-b7b5-7d996b5a193a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 03 21:36:24.182016 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:24.181979    5053 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 03 21:36:24.182211 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:24.182044    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb" podUID="9b3caedf-8d81-4f57-b7b5-7d996b5a193a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 03 21:36:25.182317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:25.182277    5053 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 03 21:36:25.182885 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:25.182354    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb" podUID="9b3caedf-8d81-4f57-b7b5-7d996b5a193a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 03 21:36:26.182430 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:26.182389    5053 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 03 21:36:26.182923 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:26.182448    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb" podUID="9b3caedf-8d81-4f57-b7b5-7d996b5a193a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 03 21:36:27.186077 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:27.186039    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:36:27.186565 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:27.186530    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:36:27.187297 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:27.187278    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:36:34.597313 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:34.597281    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95613b9c-prwz2"]
Nov 03 21:36:34.601566 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:34.601545    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95613b9c-prwz2"
Nov 03 21:36:34.660619 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:34.660576    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95613b9c-prwz2"]
Nov 03 21:36:34.703805 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:34.703759    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:36:34.703805 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:34.703757    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:36:34.703805 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:34.703761    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:36:34.703805 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:34.703810    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:36:34.709499 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:34.709476    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15905e54-b9dd-4554-9d1e-8511d866d387-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95613b9c-prwz2\" (UID: \"15905e54-b9dd-4554-9d1e-8511d866d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95613b9c-prwz2"
Nov 03 21:36:34.709617 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:34.709510    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15905e54-b9dd-4554-9d1e-8511d866d387-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95613b9c-prwz2\" (UID: \"15905e54-b9dd-4554-9d1e-8511d866d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95613b9c-prwz2"
Nov 03 21:36:34.709617 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:34.709533    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfz4s\" (UniqueName: \"kubernetes.io/projected/15905e54-b9dd-4554-9d1e-8511d866d387-kube-api-access-jfz4s\") pod \"maas-api-revoke-sub-95613b9c-prwz2\" (UID: \"15905e54-b9dd-4554-9d1e-8511d866d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95613b9c-prwz2"
Nov 03 21:36:34.811000 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:34.810970    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15905e54-b9dd-4554-9d1e-8511d866d387-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95613b9c-prwz2\" (UID: \"15905e54-b9dd-4554-9d1e-8511d866d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95613b9c-prwz2"
Nov 03 21:36:34.811205 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:34.811013    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15905e54-b9dd-4554-9d1e-8511d866d387-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95613b9c-prwz2\" (UID: \"15905e54-b9dd-4554-9d1e-8511d866d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95613b9c-prwz2"
Nov 03 21:36:34.811205 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:34.811041    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfz4s\" (UniqueName: \"kubernetes.io/projected/15905e54-b9dd-4554-9d1e-8511d866d387-kube-api-access-jfz4s\") pod \"maas-api-revoke-sub-95613b9c-prwz2\" (UID: \"15905e54-b9dd-4554-9d1e-8511d866d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95613b9c-prwz2"
Nov 03 21:36:34.811767 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:34.811738    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15905e54-b9dd-4554-9d1e-8511d866d387-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95613b9c-prwz2\" (UID: \"15905e54-b9dd-4554-9d1e-8511d866d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95613b9c-prwz2"
Nov 03 21:36:34.813586 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:34.813563    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15905e54-b9dd-4554-9d1e-8511d866d387-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95613b9c-prwz2\" (UID: \"15905e54-b9dd-4554-9d1e-8511d866d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95613b9c-prwz2"
Nov 03 21:36:34.867349 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:34.867282    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfz4s\" (UniqueName: \"kubernetes.io/projected/15905e54-b9dd-4554-9d1e-8511d866d387-kube-api-access-jfz4s\") pod \"maas-api-revoke-sub-95613b9c-prwz2\" (UID: \"15905e54-b9dd-4554-9d1e-8511d866d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95613b9c-prwz2"
Nov 03 21:36:34.918465 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:34.918438    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95613b9c-prwz2"
Nov 03 21:36:35.299060 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:36:35.299027    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15905e54_b9dd_4554_9d1e_8511d866d387.slice/crio-30fe90b7fabaf9dd7d80ba398fd9021fb91cee45ab8b2514c8facf2a986581d4 WatchSource:0}: Error finding container 30fe90b7fabaf9dd7d80ba398fd9021fb91cee45ab8b2514c8facf2a986581d4: Status 404 returned error can't find the container with id 30fe90b7fabaf9dd7d80ba398fd9021fb91cee45ab8b2514c8facf2a986581d4
Nov 03 21:36:35.299881 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:35.299856    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95613b9c-prwz2"]
Nov 03 21:36:35.643926 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:35.643888    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95613b9c-prwz2" event={"ID":"15905e54-b9dd-4554-9d1e-8511d866d387","Type":"ContainerStarted","Data":"30fe90b7fabaf9dd7d80ba398fd9021fb91cee45ab8b2514c8facf2a986581d4"}
Nov 03 21:36:36.653746 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:36.653664    5053 generic.go:356] "Generic (PLEG): container finished" podID="15905e54-b9dd-4554-9d1e-8511d866d387" containerID="4518044edfbf9a3d2b488bd86268fccbaaccd9881f7d17121dfa9752e9580eeb" exitCode=0
Nov 03 21:36:36.654127 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:36.653746    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95613b9c-prwz2" event={"ID":"15905e54-b9dd-4554-9d1e-8511d866d387","Type":"ContainerDied","Data":"4518044edfbf9a3d2b488bd86268fccbaaccd9881f7d17121dfa9752e9580eeb"}
Nov 03 21:36:37.822201 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:37.822174    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95613b9c-prwz2"
Nov 03 21:36:37.854486 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:37.854454    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15905e54-b9dd-4554-9d1e-8511d866d387-kube-api-access-jfz4s\" (UniqueName: \"kubernetes.io/projected/15905e54-b9dd-4554-9d1e-8511d866d387-kube-api-access-jfz4s\") pod \"15905e54-b9dd-4554-9d1e-8511d866d387\" (UID: \"15905e54-b9dd-4554-9d1e-8511d866d387\") "
Nov 03 21:36:37.854677 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:37.854551    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15905e54-b9dd-4554-9d1e-8511d866d387-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15905e54-b9dd-4554-9d1e-8511d866d387-maas-api-openshift-ca\") pod \"15905e54-b9dd-4554-9d1e-8511d866d387\" (UID: \"15905e54-b9dd-4554-9d1e-8511d866d387\") "
Nov 03 21:36:37.854677 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:37.854588    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15905e54-b9dd-4554-9d1e-8511d866d387-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15905e54-b9dd-4554-9d1e-8511d866d387-maas-api-xks-ca\") pod \"15905e54-b9dd-4554-9d1e-8511d866d387\" (UID: \"15905e54-b9dd-4554-9d1e-8511d866d387\") "
Nov 03 21:36:37.855009 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:37.854986    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15905e54-b9dd-4554-9d1e-8511d866d387-maas-api-openshift-ca" pod "15905e54-b9dd-4554-9d1e-8511d866d387" (UID: "15905e54-b9dd-4554-9d1e-8511d866d387"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:36:37.855178 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:37.855164    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15905e54-b9dd-4554-9d1e-8511d866d387-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:36:37.857034 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:37.857005    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15905e54-b9dd-4554-9d1e-8511d866d387-kube-api-access-jfz4s" pod "15905e54-b9dd-4554-9d1e-8511d866d387" (UID: "15905e54-b9dd-4554-9d1e-8511d866d387"). InnerVolumeSpecName "kube-api-access-jfz4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:36:37.857299 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:37.857278    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15905e54-b9dd-4554-9d1e-8511d866d387-maas-api-xks-ca" pod "15905e54-b9dd-4554-9d1e-8511d866d387" (UID: "15905e54-b9dd-4554-9d1e-8511d866d387"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:36:37.956880 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:37.956848    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfz4s\" (UniqueName: \"kubernetes.io/projected/15905e54-b9dd-4554-9d1e-8511d866d387-kube-api-access-jfz4s\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:36:37.956880 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:37.956882    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15905e54-b9dd-4554-9d1e-8511d866d387-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:36:38.678192 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:38.678163    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95613b9c-prwz2"
Nov 03 21:36:38.678192 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:38.678208    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95613b9c-prwz2" event={"ID":"15905e54-b9dd-4554-9d1e-8511d866d387","Type":"ContainerDied","Data":"30fe90b7fabaf9dd7d80ba398fd9021fb91cee45ab8b2514c8facf2a986581d4"}
Nov 03 21:36:38.678504 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:38.678301    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30fe90b7fabaf9dd7d80ba398fd9021fb91cee45ab8b2514c8facf2a986581d4"
Nov 03 21:36:40.930059 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:36:40.930025    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:36:41.631922 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:41.631884    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fad433-xg4nh"]
Nov 03 21:36:41.632648 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:41.632629    5053 cpu_manager.go:414] "containerMap: removing container" podUID="15905e54-b9dd-4554-9d1e-8511d866d387" containerName="revoke-keys"
Nov 03 21:36:41.632648 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:41.632646    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15905e54-b9dd-4554-9d1e-8511d866d387" containerName="revoke-keys"
Nov 03 21:36:41.632819 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:41.632756    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="15905e54-b9dd-4554-9d1e-8511d866d387" containerName="revoke-keys"
Nov 03 21:36:41.636853 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:41.636831    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fad433-xg4nh"
Nov 03 21:36:41.691896 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:41.691869    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fad433-xg4nh"]
Nov 03 21:36:41.720191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:41.720142    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489a5fc0-8d59-47de-8289-41120739ab55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70fad433-xg4nh\" (UID: \"489a5fc0-8d59-47de-8289-41120739ab55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fad433-xg4nh"
Nov 03 21:36:41.720392 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:41.720316    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489a5fc0-8d59-47de-8289-41120739ab55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70fad433-xg4nh\" (UID: \"489a5fc0-8d59-47de-8289-41120739ab55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fad433-xg4nh"
Nov 03 21:36:41.720470 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:41.720449    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvm9x\" (UniqueName: \"kubernetes.io/projected/489a5fc0-8d59-47de-8289-41120739ab55-kube-api-access-hvm9x\") pod \"maas-api-revoke-sub-70fad433-xg4nh\" (UID: \"489a5fc0-8d59-47de-8289-41120739ab55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fad433-xg4nh"
Nov 03 21:36:41.734429 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:41.734399    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:36:41.734785 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:41.734767    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:36:41.734892 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:41.734799    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:36:41.738091 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:41.738072    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:36:41.822306 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:41.822269    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489a5fc0-8d59-47de-8289-41120739ab55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70fad433-xg4nh\" (UID: \"489a5fc0-8d59-47de-8289-41120739ab55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fad433-xg4nh"
Nov 03 21:36:41.822480 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:41.822388    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489a5fc0-8d59-47de-8289-41120739ab55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70fad433-xg4nh\" (UID: \"489a5fc0-8d59-47de-8289-41120739ab55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fad433-xg4nh"
Nov 03 21:36:41.822559 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:41.822488    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvm9x\" (UniqueName: \"kubernetes.io/projected/489a5fc0-8d59-47de-8289-41120739ab55-kube-api-access-hvm9x\") pod \"maas-api-revoke-sub-70fad433-xg4nh\" (UID: \"489a5fc0-8d59-47de-8289-41120739ab55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fad433-xg4nh"
Nov 03 21:36:41.823029 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:41.823008    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489a5fc0-8d59-47de-8289-41120739ab55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70fad433-xg4nh\" (UID: \"489a5fc0-8d59-47de-8289-41120739ab55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fad433-xg4nh"
Nov 03 21:36:41.824830 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:41.824806    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489a5fc0-8d59-47de-8289-41120739ab55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70fad433-xg4nh\" (UID: \"489a5fc0-8d59-47de-8289-41120739ab55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fad433-xg4nh"
Nov 03 21:36:41.878981 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:41.878950    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvm9x\" (UniqueName: \"kubernetes.io/projected/489a5fc0-8d59-47de-8289-41120739ab55-kube-api-access-hvm9x\") pod \"maas-api-revoke-sub-70fad433-xg4nh\" (UID: \"489a5fc0-8d59-47de-8289-41120739ab55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fad433-xg4nh"
Nov 03 21:36:41.952689 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:41.952658    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fad433-xg4nh"
Nov 03 21:36:42.330423 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:36:42.330391    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489a5fc0_8d59_47de_8289_41120739ab55.slice/crio-ea9c07e485b8e0909a5b44be97e9ec8813ab6c3f852289ce49fa71066ea0eb70 WatchSource:0}: Error finding container ea9c07e485b8e0909a5b44be97e9ec8813ab6c3f852289ce49fa71066ea0eb70: Status 404 returned error can't find the container with id ea9c07e485b8e0909a5b44be97e9ec8813ab6c3f852289ce49fa71066ea0eb70
Nov 03 21:36:42.330568 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:42.330461    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fad433-xg4nh"]
Nov 03 21:36:42.732257 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:42.729154    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fad433-xg4nh" event={"ID":"489a5fc0-8d59-47de-8289-41120739ab55","Type":"ContainerStarted","Data":"81f83ba8c529028aec9e4bf2f85aa5df43924bd8edd9d6b69c8af737e74b5cc3"}
Nov 03 21:36:42.732257 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:42.729318    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fad433-xg4nh" event={"ID":"489a5fc0-8d59-47de-8289-41120739ab55","Type":"ContainerStarted","Data":"ea9c07e485b8e0909a5b44be97e9ec8813ab6c3f852289ce49fa71066ea0eb70"}
Nov 03 21:36:42.847726 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:42.847677    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fad433-xg4nh" podStartSLOduration=1.8476628609999999 podStartE2EDuration="1.847662861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:36:42.843409491 +0000 UTC m=+1317.337331366" watchObservedRunningTime="2026-11-03 21:36:42.847662861 +0000 UTC m=+1317.341584736"
Nov 03 21:36:43.358975 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:36:43.358933    5053 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489a5fc0_8d59_47de_8289_41120739ab55.slice/crio-conmon-81f83ba8c529028aec9e4bf2f85aa5df43924bd8edd9d6b69c8af737e74b5cc3.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 21:36:43.740441 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:43.740407    5053 generic.go:356] "Generic (PLEG): container finished" podID="489a5fc0-8d59-47de-8289-41120739ab55" containerID="81f83ba8c529028aec9e4bf2f85aa5df43924bd8edd9d6b69c8af737e74b5cc3" exitCode=0
Nov 03 21:36:43.740674 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:43.740491    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fad433-xg4nh" event={"ID":"489a5fc0-8d59-47de-8289-41120739ab55","Type":"ContainerDied","Data":"81f83ba8c529028aec9e4bf2f85aa5df43924bd8edd9d6b69c8af737e74b5cc3"}
Nov 03 21:36:44.911896 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:44.911866    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fad433-xg4nh"
Nov 03 21:36:45.076843 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:45.076738    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/489a5fc0-8d59-47de-8289-41120739ab55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489a5fc0-8d59-47de-8289-41120739ab55-maas-api-xks-ca\") pod \"489a5fc0-8d59-47de-8289-41120739ab55\" (UID: \"489a5fc0-8d59-47de-8289-41120739ab55\") "
Nov 03 21:36:45.077025 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:45.076903    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/489a5fc0-8d59-47de-8289-41120739ab55-kube-api-access-hvm9x\" (UniqueName: \"kubernetes.io/projected/489a5fc0-8d59-47de-8289-41120739ab55-kube-api-access-hvm9x\") pod \"489a5fc0-8d59-47de-8289-41120739ab55\" (UID: \"489a5fc0-8d59-47de-8289-41120739ab55\") "
Nov 03 21:36:45.077025 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:45.076938    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/489a5fc0-8d59-47de-8289-41120739ab55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489a5fc0-8d59-47de-8289-41120739ab55-maas-api-openshift-ca\") pod \"489a5fc0-8d59-47de-8289-41120739ab55\" (UID: \"489a5fc0-8d59-47de-8289-41120739ab55\") "
Nov 03 21:36:45.077383 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:45.077356    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/489a5fc0-8d59-47de-8289-41120739ab55-maas-api-openshift-ca" pod "489a5fc0-8d59-47de-8289-41120739ab55" (UID: "489a5fc0-8d59-47de-8289-41120739ab55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:36:45.077805 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:45.077784    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489a5fc0-8d59-47de-8289-41120739ab55-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:36:45.079451 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:45.079424    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/489a5fc0-8d59-47de-8289-41120739ab55-maas-api-xks-ca" pod "489a5fc0-8d59-47de-8289-41120739ab55" (UID: "489a5fc0-8d59-47de-8289-41120739ab55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:36:45.080159 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:45.080135    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489a5fc0-8d59-47de-8289-41120739ab55-kube-api-access-hvm9x" pod "489a5fc0-8d59-47de-8289-41120739ab55" (UID: "489a5fc0-8d59-47de-8289-41120739ab55"). InnerVolumeSpecName "kube-api-access-hvm9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:36:45.179094 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:45.179050    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489a5fc0-8d59-47de-8289-41120739ab55-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:36:45.179094 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:45.179085    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvm9x\" (UniqueName: \"kubernetes.io/projected/489a5fc0-8d59-47de-8289-41120739ab55-kube-api-access-hvm9x\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:36:45.759300 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:45.759273    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fad433-xg4nh"
Nov 03 21:36:45.759476 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:45.759269    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fad433-xg4nh" event={"ID":"489a5fc0-8d59-47de-8289-41120739ab55","Type":"ContainerDied","Data":"ea9c07e485b8e0909a5b44be97e9ec8813ab6c3f852289ce49fa71066ea0eb70"}
Nov 03 21:36:45.759476 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:45.759399    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea9c07e485b8e0909a5b44be97e9ec8813ab6c3f852289ce49fa71066ea0eb70"
Nov 03 21:36:46.124259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:46.120043    5053 scope.go:122] "RemoveContainer" containerID="328d6ce386683033545bf1d052df3f9bf1825fcc4fd5bb1812a53faa8ddee48f"
Nov 03 21:36:46.139104 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:46.139077    5053 scope.go:122] "RemoveContainer" containerID="0096b3c5e66025f031095cdec7ad986e98128d73989981e5b61fec71b018187f"
Nov 03 21:36:46.150498 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:46.150476    5053 scope.go:122] "RemoveContainer" containerID="7d116c38cbee2d838656a0a4d83495f2c8e68dc97f1a77f34b5edb9753fcf8f1"
Nov 03 21:36:46.159870 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:46.159848    5053 scope.go:122] "RemoveContainer" containerID="7fd955a21c0de66ceb8e93c7cb0a74089d84589fec1e4a747f9b54bfccb5de28"
Nov 03 21:36:46.169187 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:46.169163    5053 scope.go:122] "RemoveContainer" containerID="bef82bb3e5ee4eb7a6b369f41cd030a0d503687e94e0b0c3f205fdd363e8a0a5"
Nov 03 21:36:49.670261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:49.656331    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx" podUID="a6c0d3aa-18c2-465b-846b-b82968acc780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 03 21:36:50.820593 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:50.820553    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1db22669-bvpzd"]
Nov 03 21:36:50.821485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:50.821463    5053 cpu_manager.go:414] "containerMap: removing container" podUID="489a5fc0-8d59-47de-8289-41120739ab55" containerName="revoke-keys"
Nov 03 21:36:50.821485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:50.821486    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="489a5fc0-8d59-47de-8289-41120739ab55" containerName="revoke-keys"
Nov 03 21:36:50.821705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:50.821691    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="489a5fc0-8d59-47de-8289-41120739ab55" containerName="revoke-keys"
Nov 03 21:36:50.826757 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:50.826734    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db22669-bvpzd"
Nov 03 21:36:50.881593 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:50.881557    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1db22669-bvpzd"]
Nov 03 21:36:50.924650 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:50.924617    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:36:50.924650 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:50.924627    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:36:50.924650 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:50.924627    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:36:50.928881 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:50.928525    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:36:50.963770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:50.963739    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmndt\" (UniqueName: \"kubernetes.io/projected/c4958759-1aab-42ad-b08b-263aa890a359-kube-api-access-jmndt\") pod \"maas-api-revoke-sub-1db22669-bvpzd\" (UID: \"c4958759-1aab-42ad-b08b-263aa890a359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db22669-bvpzd"
Nov 03 21:36:50.963770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:50.963782    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4958759-1aab-42ad-b08b-263aa890a359-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1db22669-bvpzd\" (UID: \"c4958759-1aab-42ad-b08b-263aa890a359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db22669-bvpzd"
Nov 03 21:36:50.963978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:50.963811    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4958759-1aab-42ad-b08b-263aa890a359-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1db22669-bvpzd\" (UID: \"c4958759-1aab-42ad-b08b-263aa890a359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db22669-bvpzd"
Nov 03 21:36:51.065367 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:51.065329    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4958759-1aab-42ad-b08b-263aa890a359-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1db22669-bvpzd\" (UID: \"c4958759-1aab-42ad-b08b-263aa890a359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db22669-bvpzd"
Nov 03 21:36:51.065548 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:51.065387    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4958759-1aab-42ad-b08b-263aa890a359-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1db22669-bvpzd\" (UID: \"c4958759-1aab-42ad-b08b-263aa890a359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db22669-bvpzd"
Nov 03 21:36:51.065593 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:51.065574    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmndt\" (UniqueName: \"kubernetes.io/projected/c4958759-1aab-42ad-b08b-263aa890a359-kube-api-access-jmndt\") pod \"maas-api-revoke-sub-1db22669-bvpzd\" (UID: \"c4958759-1aab-42ad-b08b-263aa890a359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db22669-bvpzd"
Nov 03 21:36:51.066192 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:51.066114    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4958759-1aab-42ad-b08b-263aa890a359-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1db22669-bvpzd\" (UID: \"c4958759-1aab-42ad-b08b-263aa890a359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db22669-bvpzd"
Nov 03 21:36:51.067859 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:51.067835    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4958759-1aab-42ad-b08b-263aa890a359-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1db22669-bvpzd\" (UID: \"c4958759-1aab-42ad-b08b-263aa890a359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db22669-bvpzd"
Nov 03 21:36:51.126869 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:51.126794    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmndt\" (UniqueName: \"kubernetes.io/projected/c4958759-1aab-42ad-b08b-263aa890a359-kube-api-access-jmndt\") pod \"maas-api-revoke-sub-1db22669-bvpzd\" (UID: \"c4958759-1aab-42ad-b08b-263aa890a359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db22669-bvpzd"
Nov 03 21:36:51.143668 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:51.143635    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db22669-bvpzd"
Nov 03 21:36:51.452504 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:51.452471    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17c89b26-wcx94"]
Nov 03 21:36:51.458128 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:51.458110    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c89b26-wcx94"
Nov 03 21:36:51.513643 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:51.513607    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17c89b26-wcx94"]
Nov 03 21:36:51.562688 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:36:51.562654    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4958759_1aab_42ad_b08b_263aa890a359.slice/crio-a84feb578bb8e4131de835aa68a55cf1a96387e1b30adf75d67b15b9fc798424 WatchSource:0}: Error finding container a84feb578bb8e4131de835aa68a55cf1a96387e1b30adf75d67b15b9fc798424: Status 404 returned error can't find the container with id a84feb578bb8e4131de835aa68a55cf1a96387e1b30adf75d67b15b9fc798424
Nov 03 21:36:51.563172 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:51.563144    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1db22669-bvpzd"]
Nov 03 21:36:51.573558 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:51.573523    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ad4db01-376c-40c2-966a-2f764b23fa5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17c89b26-wcx94\" (UID: \"1ad4db01-376c-40c2-966a-2f764b23fa5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c89b26-wcx94"
Nov 03 21:36:51.573680 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:51.573580    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fh2h\" (UniqueName: \"kubernetes.io/projected/1ad4db01-376c-40c2-966a-2f764b23fa5f-kube-api-access-4fh2h\") pod \"maas-api-revoke-sub-17c89b26-wcx94\" (UID: \"1ad4db01-376c-40c2-966a-2f764b23fa5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c89b26-wcx94"
Nov 03 21:36:51.573739 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:51.573721    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ad4db01-376c-40c2-966a-2f764b23fa5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17c89b26-wcx94\" (UID: \"1ad4db01-376c-40c2-966a-2f764b23fa5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c89b26-wcx94"
Nov 03 21:36:51.674935 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:51.674904    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fh2h\" (UniqueName: \"kubernetes.io/projected/1ad4db01-376c-40c2-966a-2f764b23fa5f-kube-api-access-4fh2h\") pod \"maas-api-revoke-sub-17c89b26-wcx94\" (UID: \"1ad4db01-376c-40c2-966a-2f764b23fa5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c89b26-wcx94"
Nov 03 21:36:51.675108 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:51.674993    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ad4db01-376c-40c2-966a-2f764b23fa5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17c89b26-wcx94\" (UID: \"1ad4db01-376c-40c2-966a-2f764b23fa5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c89b26-wcx94"
Nov 03 21:36:51.675158 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:51.675144    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ad4db01-376c-40c2-966a-2f764b23fa5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17c89b26-wcx94\" (UID: \"1ad4db01-376c-40c2-966a-2f764b23fa5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c89b26-wcx94"
Nov 03 21:36:51.675758 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:51.675726    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ad4db01-376c-40c2-966a-2f764b23fa5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17c89b26-wcx94\" (UID: \"1ad4db01-376c-40c2-966a-2f764b23fa5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c89b26-wcx94"
Nov 03 21:36:51.677433 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:51.677416    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ad4db01-376c-40c2-966a-2f764b23fa5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17c89b26-wcx94\" (UID: \"1ad4db01-376c-40c2-966a-2f764b23fa5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c89b26-wcx94"
Nov 03 21:36:51.734865 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:51.734825    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fh2h\" (UniqueName: \"kubernetes.io/projected/1ad4db01-376c-40c2-966a-2f764b23fa5f-kube-api-access-4fh2h\") pod \"maas-api-revoke-sub-17c89b26-wcx94\" (UID: \"1ad4db01-376c-40c2-966a-2f764b23fa5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c89b26-wcx94"
Nov 03 21:36:51.772380 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:51.772333    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c89b26-wcx94"
Nov 03 21:36:51.860866 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:51.860825    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db22669-bvpzd" event={"ID":"c4958759-1aab-42ad-b08b-263aa890a359","Type":"ContainerStarted","Data":"a84feb578bb8e4131de835aa68a55cf1a96387e1b30adf75d67b15b9fc798424"}
Nov 03 21:36:52.153417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:52.153389    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17c89b26-wcx94"]
Nov 03 21:36:52.153529 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:36:52.153412    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad4db01_376c_40c2_966a_2f764b23fa5f.slice/crio-93beb4362e9236acdac929c2a73fb24f3361c260b44052536598917139ecf1f7 WatchSource:0}: Error finding container 93beb4362e9236acdac929c2a73fb24f3361c260b44052536598917139ecf1f7: Status 404 returned error can't find the container with id 93beb4362e9236acdac929c2a73fb24f3361c260b44052536598917139ecf1f7
Nov 03 21:36:52.870124 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:52.870018    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c89b26-wcx94" event={"ID":"1ad4db01-376c-40c2-966a-2f764b23fa5f","Type":"ContainerStarted","Data":"f3d042f71bb4a3a64071dab7577ce8575934493d2f020204b023cb581af05365"}
Nov 03 21:36:52.870124 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:52.870088    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c89b26-wcx94" event={"ID":"1ad4db01-376c-40c2-966a-2f764b23fa5f","Type":"ContainerStarted","Data":"93beb4362e9236acdac929c2a73fb24f3361c260b44052536598917139ecf1f7"}
Nov 03 21:36:52.871779 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:52.871749    5053 generic.go:356] "Generic (PLEG): container finished" podID="c4958759-1aab-42ad-b08b-263aa890a359" containerID="c8b33489f62be017180f12303f6539dc7e80ca49b4dd77d33dd3ab1a33d5d0d6" exitCode=0
Nov 03 21:36:52.871902 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:52.871845    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db22669-bvpzd" event={"ID":"c4958759-1aab-42ad-b08b-263aa890a359","Type":"ContainerDied","Data":"c8b33489f62be017180f12303f6539dc7e80ca49b4dd77d33dd3ab1a33d5d0d6"}
Nov 03 21:36:52.983966 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:52.983905    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c89b26-wcx94" podStartSLOduration=1.983887385 podStartE2EDuration="1.983887385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:36:52.983163573 +0000 UTC m=+1327.477085448" watchObservedRunningTime="2026-11-03 21:36:52.983887385 +0000 UTC m=+1327.477809260"
Nov 03 21:36:53.885942 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:53.885905    5053 generic.go:356] "Generic (PLEG): container finished" podID="1ad4db01-376c-40c2-966a-2f764b23fa5f" containerID="f3d042f71bb4a3a64071dab7577ce8575934493d2f020204b023cb581af05365" exitCode=0
Nov 03 21:36:53.885942 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:53.885936    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c89b26-wcx94" event={"ID":"1ad4db01-376c-40c2-966a-2f764b23fa5f","Type":"ContainerDied","Data":"f3d042f71bb4a3a64071dab7577ce8575934493d2f020204b023cb581af05365"}
Nov 03 21:36:54.076966 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:54.076939    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db22669-bvpzd"
Nov 03 21:36:54.236867 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:54.236829    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4958759-1aab-42ad-b08b-263aa890a359-kube-api-access-jmndt\" (UniqueName: \"kubernetes.io/projected/c4958759-1aab-42ad-b08b-263aa890a359-kube-api-access-jmndt\") pod \"c4958759-1aab-42ad-b08b-263aa890a359\" (UID: \"c4958759-1aab-42ad-b08b-263aa890a359\") "
Nov 03 21:36:54.237074 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:54.236952    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4958759-1aab-42ad-b08b-263aa890a359-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4958759-1aab-42ad-b08b-263aa890a359-maas-api-openshift-ca\") pod \"c4958759-1aab-42ad-b08b-263aa890a359\" (UID: \"c4958759-1aab-42ad-b08b-263aa890a359\") "
Nov 03 21:36:54.237136 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:54.237082    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4958759-1aab-42ad-b08b-263aa890a359-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4958759-1aab-42ad-b08b-263aa890a359-maas-api-xks-ca\") pod \"c4958759-1aab-42ad-b08b-263aa890a359\" (UID: \"c4958759-1aab-42ad-b08b-263aa890a359\") "
Nov 03 21:36:54.237960 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:54.237934    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4958759-1aab-42ad-b08b-263aa890a359-maas-api-openshift-ca" pod "c4958759-1aab-42ad-b08b-263aa890a359" (UID: "c4958759-1aab-42ad-b08b-263aa890a359"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:36:54.239679 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:54.239650    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4958759-1aab-42ad-b08b-263aa890a359-maas-api-xks-ca" pod "c4958759-1aab-42ad-b08b-263aa890a359" (UID: "c4958759-1aab-42ad-b08b-263aa890a359"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:36:54.240955 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:54.240935    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4958759-1aab-42ad-b08b-263aa890a359-kube-api-access-jmndt" pod "c4958759-1aab-42ad-b08b-263aa890a359" (UID: "c4958759-1aab-42ad-b08b-263aa890a359"). InnerVolumeSpecName "kube-api-access-jmndt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:36:54.338887 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:54.338850    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmndt\" (UniqueName: \"kubernetes.io/projected/c4958759-1aab-42ad-b08b-263aa890a359-kube-api-access-jmndt\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:36:54.338887 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:54.338881    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4958759-1aab-42ad-b08b-263aa890a359-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:36:54.338887 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:54.338891    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4958759-1aab-42ad-b08b-263aa890a359-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:36:54.895717 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:54.895686    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db22669-bvpzd"
Nov 03 21:36:54.896137 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:54.895712    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db22669-bvpzd" event={"ID":"c4958759-1aab-42ad-b08b-263aa890a359","Type":"ContainerDied","Data":"a84feb578bb8e4131de835aa68a55cf1a96387e1b30adf75d67b15b9fc798424"}
Nov 03 21:36:54.896137 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:54.895767    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a84feb578bb8e4131de835aa68a55cf1a96387e1b30adf75d67b15b9fc798424"
Nov 03 21:36:55.059942 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:55.059915    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c89b26-wcx94"
Nov 03 21:36:55.251964 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:55.251936    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ad4db01-376c-40c2-966a-2f764b23fa5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ad4db01-376c-40c2-966a-2f764b23fa5f-maas-api-openshift-ca\") pod \"1ad4db01-376c-40c2-966a-2f764b23fa5f\" (UID: \"1ad4db01-376c-40c2-966a-2f764b23fa5f\") "
Nov 03 21:36:55.252136 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:55.252117    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ad4db01-376c-40c2-966a-2f764b23fa5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ad4db01-376c-40c2-966a-2f764b23fa5f-maas-api-xks-ca\") pod \"1ad4db01-376c-40c2-966a-2f764b23fa5f\" (UID: \"1ad4db01-376c-40c2-966a-2f764b23fa5f\") "
Nov 03 21:36:55.252180 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:55.252145    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ad4db01-376c-40c2-966a-2f764b23fa5f-kube-api-access-4fh2h\" (UniqueName: \"kubernetes.io/projected/1ad4db01-376c-40c2-966a-2f764b23fa5f-kube-api-access-4fh2h\") pod \"1ad4db01-376c-40c2-966a-2f764b23fa5f\" (UID: \"1ad4db01-376c-40c2-966a-2f764b23fa5f\") "
Nov 03 21:36:55.252296 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:55.252276    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ad4db01-376c-40c2-966a-2f764b23fa5f-maas-api-openshift-ca" pod "1ad4db01-376c-40c2-966a-2f764b23fa5f" (UID: "1ad4db01-376c-40c2-966a-2f764b23fa5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:36:55.252703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:55.252671    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ad4db01-376c-40c2-966a-2f764b23fa5f-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:36:55.254341 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:55.254322    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ad4db01-376c-40c2-966a-2f764b23fa5f-maas-api-xks-ca" pod "1ad4db01-376c-40c2-966a-2f764b23fa5f" (UID: "1ad4db01-376c-40c2-966a-2f764b23fa5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:36:55.254463 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:55.254449    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad4db01-376c-40c2-966a-2f764b23fa5f-kube-api-access-4fh2h" pod "1ad4db01-376c-40c2-966a-2f764b23fa5f" (UID: "1ad4db01-376c-40c2-966a-2f764b23fa5f"). InnerVolumeSpecName "kube-api-access-4fh2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:36:55.353781 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:55.353751    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ad4db01-376c-40c2-966a-2f764b23fa5f-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:36:55.353781 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:55.353787    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fh2h\" (UniqueName: \"kubernetes.io/projected/1ad4db01-376c-40c2-966a-2f764b23fa5f-kube-api-access-4fh2h\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:36:55.905461 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:55.905430    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c89b26-wcx94"
Nov 03 21:36:55.905840 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:55.905472    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c89b26-wcx94" event={"ID":"1ad4db01-376c-40c2-966a-2f764b23fa5f","Type":"ContainerDied","Data":"93beb4362e9236acdac929c2a73fb24f3361c260b44052536598917139ecf1f7"}
Nov 03 21:36:55.905840 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:36:55.905522    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93beb4362e9236acdac929c2a73fb24f3361c260b44052536598917139ecf1f7"
Nov 03 21:37:03.558342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.558311    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"]
Nov 03 21:37:03.558991 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.558977    5053 cpu_manager.go:414] "containerMap: removing container" podUID="c4958759-1aab-42ad-b08b-263aa890a359" containerName="revoke-keys"
Nov 03 21:37:03.559057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.558993    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4958759-1aab-42ad-b08b-263aa890a359" containerName="revoke-keys"
Nov 03 21:37:03.559057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.559002    5053 cpu_manager.go:414] "containerMap: removing container" podUID="1ad4db01-376c-40c2-966a-2f764b23fa5f" containerName="revoke-keys"
Nov 03 21:37:03.559057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.559007    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ad4db01-376c-40c2-966a-2f764b23fa5f" containerName="revoke-keys"
Nov 03 21:37:03.559146 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.559137    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4958759-1aab-42ad-b08b-263aa890a359" containerName="revoke-keys"
Nov 03 21:37:03.559146 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.559144    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ad4db01-376c-40c2-966a-2f764b23fa5f" containerName="revoke-keys"
Nov 03 21:37:03.563000 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.562983    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.667750 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.667712    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-credential-socket\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.667948 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.667927    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-workload-socket\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.668024 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.667967    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b508d7b-1822-47fa-9565-61da014df9c3-istiod-ca-cert\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.668087 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.668043    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-istio-envoy\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.668087 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.668074    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b508d7b-1822-47fa-9565-61da014df9c3-istio-token\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.668187 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.668129    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xngk7\" (UniqueName: \"kubernetes.io/projected/7b508d7b-1822-47fa-9565-61da014df9c3-kube-api-access-xngk7\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.668275 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.668183    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-istio-data\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.668341 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.668267    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b508d7b-1822-47fa-9565-61da014df9c3-istio-podinfo\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.668397 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.668360    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-workload-certs\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.672040 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.672011    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"]
Nov 03 21:37:03.707596 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.707554    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-29472d-openshift-default-dockercfg-m87gh\""
Nov 03 21:37:03.770154 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.770124    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-istio-envoy\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.770154 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.770162    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b508d7b-1822-47fa-9565-61da014df9c3-istio-token\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.770460 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.770213    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xngk7\" (UniqueName: \"kubernetes.io/projected/7b508d7b-1822-47fa-9565-61da014df9c3-kube-api-access-xngk7\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.770460 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.770277    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-istio-data\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.770460 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.770412    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b508d7b-1822-47fa-9565-61da014df9c3-istio-podinfo\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.770617 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.770496    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-workload-certs\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.770617 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.770527    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-credential-socket\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.770725 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.770672    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-workload-socket\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.770725 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.770706    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b508d7b-1822-47fa-9565-61da014df9c3-istiod-ca-cert\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.770934 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.770908    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-istio-data\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.774584 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.771065    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-workload-certs\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.774584 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.771508    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b508d7b-1822-47fa-9565-61da014df9c3-istiod-ca-cert\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.774584 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.771676    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-credential-socket\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.774584 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.772106    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-workload-socket\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.774584 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.774157    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b508d7b-1822-47fa-9565-61da014df9c3-istio-podinfo\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.778913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.778885    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-istio-envoy\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.827695 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.827606    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b508d7b-1822-47fa-9565-61da014df9c3-istio-token\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.827985 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.827731    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xngk7\" (UniqueName: \"kubernetes.io/projected/7b508d7b-1822-47fa-9565-61da014df9c3-kube-api-access-xngk7\") pod \"e2e-api-a-29472d-openshift-default-55bd446969-f54r9\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") " pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:03.884444 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:03.884342    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:04.387870 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:37:04.387844    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b508d7b_1822_47fa_9565_61da014df9c3.slice/crio-f8d1fa4ec13856b98b2067c186ca6496bfb06d9faf9f8d3c9a9c7b1804b61b97 WatchSource:0}: Error finding container f8d1fa4ec13856b98b2067c186ca6496bfb06d9faf9f8d3c9a9c7b1804b61b97: Status 404 returned error can't find the container with id f8d1fa4ec13856b98b2067c186ca6496bfb06d9faf9f8d3c9a9c7b1804b61b97
Nov 03 21:37:04.388891 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:04.388868    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"]
Nov 03 21:37:04.390317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:04.390284    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:37:04.390402 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:04.390357    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:37:04.390457 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:04.390400    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:37:05.029260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:05.021812    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9" event={"ID":"7b508d7b-1822-47fa-9565-61da014df9c3","Type":"ContainerStarted","Data":"8b1315fa3b93f0a626aab55322e8fe8ac790bd1e711f4cbdfca21488ad0e7e3d"}
Nov 03 21:37:05.029260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:05.021880    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9" event={"ID":"7b508d7b-1822-47fa-9565-61da014df9c3","Type":"ContainerStarted","Data":"f8d1fa4ec13856b98b2067c186ca6496bfb06d9faf9f8d3c9a9c7b1804b61b97"}
Nov 03 21:37:05.189410 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:05.189356    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9" podStartSLOduration=2.189341405 podStartE2EDuration="2.189341405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:05.187348622 +0000 UTC m=+1339.681270494" watchObservedRunningTime="2026-11-03 21:37:05.189341405 +0000 UTC m=+1339.683263280"
Nov 03 21:37:05.886981 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:05.886940    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:05.893112 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:05.893082    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:06.036879 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:06.036847    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:06.037708 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:06.037673    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:37:10.928992 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:37:10.928950    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:37:11.741134 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:11.741105    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs"]
Nov 03 21:37:11.748251 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:11.748217    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs"
Nov 03 21:37:11.804083 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:11.804037    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs"]
Nov 03 21:37:11.886436 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:11.886361    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-29472d\""
Nov 03 21:37:11.932849 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:11.922362    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7650dde2-539f-4535-b69f-8e90820be009-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs\" (UID: \"7650dde2-539f-4535-b69f-8e90820be009\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs"
Nov 03 21:37:11.932849 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:11.922550    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcddc\" (UniqueName: \"kubernetes.io/projected/7650dde2-539f-4535-b69f-8e90820be009-kube-api-access-mcddc\") pod \"payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs\" (UID: \"7650dde2-539f-4535-b69f-8e90820be009\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs"
Nov 03 21:37:11.958262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:11.937894    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-29472d-dockercfg-rswnl\""
Nov 03 21:37:12.024865 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:12.024682    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcddc\" (UniqueName: \"kubernetes.io/projected/7650dde2-539f-4535-b69f-8e90820be009-kube-api-access-mcddc\") pod \"payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs\" (UID: \"7650dde2-539f-4535-b69f-8e90820be009\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs"
Nov 03 21:37:12.025399 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:12.025359    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7650dde2-539f-4535-b69f-8e90820be009-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs\" (UID: \"7650dde2-539f-4535-b69f-8e90820be009\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs"
Nov 03 21:37:12.026556 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:12.026525    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7650dde2-539f-4535-b69f-8e90820be009-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs\" (UID: \"7650dde2-539f-4535-b69f-8e90820be009\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs"
Nov 03 21:37:12.033478 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:12.033448    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr"]
Nov 03 21:37:12.039479 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:12.039450    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr"
Nov 03 21:37:12.045827 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:12.045805    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr"]
Nov 03 21:37:12.111985 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:12.111951    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcddc\" (UniqueName: \"kubernetes.io/projected/7650dde2-539f-4535-b69f-8e90820be009-kube-api-access-mcddc\") pod \"payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs\" (UID: \"7650dde2-539f-4535-b69f-8e90820be009\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs"
Nov 03 21:37:12.128351 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:12.128310    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w9sz\" (UniqueName: \"kubernetes.io/projected/e13ebd3d-b7a9-43d9-a490-4e6d37b573d7-kube-api-access-6w9sz\") pod \"payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr\" (UID: \"e13ebd3d-b7a9-43d9-a490-4e6d37b573d7\") " pod="openshift-ingress/payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr"
Nov 03 21:37:12.128600 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:12.128578    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13ebd3d-b7a9-43d9-a490-4e6d37b573d7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr\" (UID: \"e13ebd3d-b7a9-43d9-a490-4e6d37b573d7\") " pod="openshift-ingress/payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr"
Nov 03 21:37:12.229927 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:12.229896    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w9sz\" (UniqueName: \"kubernetes.io/projected/e13ebd3d-b7a9-43d9-a490-4e6d37b573d7-kube-api-access-6w9sz\") pod \"payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr\" (UID: \"e13ebd3d-b7a9-43d9-a490-4e6d37b573d7\") " pod="openshift-ingress/payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr"
Nov 03 21:37:12.230107 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:12.230071    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13ebd3d-b7a9-43d9-a490-4e6d37b573d7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr\" (UID: \"e13ebd3d-b7a9-43d9-a490-4e6d37b573d7\") " pod="openshift-ingress/payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr"
Nov 03 21:37:12.230797 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:12.230777    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13ebd3d-b7a9-43d9-a490-4e6d37b573d7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr\" (UID: \"e13ebd3d-b7a9-43d9-a490-4e6d37b573d7\") " pod="openshift-ingress/payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr"
Nov 03 21:37:12.350321 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:12.334590    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w9sz\" (UniqueName: \"kubernetes.io/projected/e13ebd3d-b7a9-43d9-a490-4e6d37b573d7-kube-api-access-6w9sz\") pod \"payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr\" (UID: \"e13ebd3d-b7a9-43d9-a490-4e6d37b573d7\") " pod="openshift-ingress/payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr"
Nov 03 21:37:12.365260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:12.362522    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr"
Nov 03 21:37:12.365260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:12.363043    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs"
Nov 03 21:37:13.071478 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.069807    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs"]
Nov 03 21:37:13.113721 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.113489    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr"]
Nov 03 21:37:13.131842 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.131821    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs" event={"ID":"7650dde2-539f-4535-b69f-8e90820be009","Type":"ContainerStarted","Data":"8fbcc6a001b27277158431c42e1af668a6425f02b4516e0a43ac3270f584a215"}
Nov 03 21:37:13.168903 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.168869    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"]
Nov 03 21:37:13.174963 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.174913    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.280419 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.280379    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"]
Nov 03 21:37:13.293819 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.293781    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.293819 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.293818    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbc98\" (UniqueName: \"kubernetes.io/projected/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-kube-api-access-nbc98\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.294066 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.293906    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.294066 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.293939    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.294066 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.293962    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.294066 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.294002    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.294406 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.294089    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.294406 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.294113    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.294406 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.294174    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.395377 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.395336    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.395377 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.395377    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.395654 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.395433    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.395654 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.395484    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.395654 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.395514    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbc98\" (UniqueName: \"kubernetes.io/projected/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-kube-api-access-nbc98\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.395654 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.395549    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.395654 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.395623    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.396113 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.395663    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.396113 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.395724    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.396113 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.395898    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.397363 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.396115    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.397363 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.396278    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.397363 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.396304    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.397363 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.397314    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.398372 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.398345    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.398680 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.398662    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.468778 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.468745    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.469832 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.469808    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbc98\" (UniqueName: \"kubernetes.io/projected/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-kube-api-access-nbc98\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.550862 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.550772    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:13.989198 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:37:13.989171    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf2bd9c_1e8e_41eb_b19f_bf150ee8e344.slice/crio-87ca2dcddeef7d6bfc63c7637e49a11a2f52b82ae660598236a3fdf8f4f62556 WatchSource:0}: Error finding container 87ca2dcddeef7d6bfc63c7637e49a11a2f52b82ae660598236a3fdf8f4f62556: Status 404 returned error can't find the container with id 87ca2dcddeef7d6bfc63c7637e49a11a2f52b82ae660598236a3fdf8f4f62556
Nov 03 21:37:13.990362 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.990335    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"]
Nov 03 21:37:13.991981 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.991951    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:37:13.992052 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.992013    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:37:13.992052 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:13.992041    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:37:14.151260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:14.150708    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs" event={"ID":"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344","Type":"ContainerStarted","Data":"55cfe9759b5da932271ca0a43b7092fc4b245a3a3f5f68f60ac730d93a6c7985"}
Nov 03 21:37:14.151260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:14.150779    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs" event={"ID":"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344","Type":"ContainerStarted","Data":"87ca2dcddeef7d6bfc63c7637e49a11a2f52b82ae660598236a3fdf8f4f62556"}
Nov 03 21:37:14.159411 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:14.154179    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr" event={"ID":"e13ebd3d-b7a9-43d9-a490-4e6d37b573d7","Type":"ContainerStarted","Data":"16dfd38fc8d926a39f12386f57f0384e7cb2d2ed8b2d2357b4e63a089e998703"}
Nov 03 21:37:14.159411 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:14.154296    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr" event={"ID":"e13ebd3d-b7a9-43d9-a490-4e6d37b573d7","Type":"ContainerStarted","Data":"8c2d5c40c7f1355cefc14e48bb87de6fd2377bd5a5425fb54f97a053857c82a8"}
Nov 03 21:37:14.159411 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:14.155035    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr"
Nov 03 21:37:14.159411 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:14.157269    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs" event={"ID":"7650dde2-539f-4535-b69f-8e90820be009","Type":"ContainerStarted","Data":"eefb5077bd928545ce3514f34f6f050316ba9ed2a6596520c0e264472e342555"}
Nov 03 21:37:14.159411 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:14.157934    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs"
Nov 03 21:37:14.269623 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:14.269506    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs" podStartSLOduration=1.269491847 podStartE2EDuration="1.269491847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:14.269099436 +0000 UTC m=+1348.763021312" watchObservedRunningTime="2026-11-03 21:37:14.269491847 +0000 UTC m=+1348.763413778"
Nov 03 21:37:14.386705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:14.386646    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs" podStartSLOduration=3.386630122 podStartE2EDuration="3.386630122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:14.382217972 +0000 UTC m=+1348.876139873" watchObservedRunningTime="2026-11-03 21:37:14.386630122 +0000 UTC m=+1348.880552028"
Nov 03 21:37:14.501749 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:14.501631    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr" podStartSLOduration=3.5016162680000003 podStartE2EDuration="3.501616268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:14.497977661 +0000 UTC m=+1348.991899535" watchObservedRunningTime="2026-11-03 21:37:14.501616268 +0000 UTC m=+1348.995538142"
Nov 03 21:37:14.564190 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:14.552347    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:15.553003 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:15.552971    5053 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 03 21:37:15.553512 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:15.553030    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs" podUID="1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 03 21:37:16.551323 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:16.551288    5053 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 03 21:37:16.551534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:16.551361    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs" podUID="1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 03 21:37:17.133260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.132327    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2"]
Nov 03 21:37:17.158702 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.157600    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2"
Nov 03 21:37:17.180424 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.180085    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4887\" (UniqueName: \"kubernetes.io/projected/57c5d5d3-f260-46e3-9ea8-75d631974635-kube-api-access-g4887\") pod \"payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2\" (UID: \"57c5d5d3-f260-46e3-9ea8-75d631974635\") " pod="openshift-ingress/payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2"
Nov 03 21:37:17.180424 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.180187    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57c5d5d3-f260-46e3-9ea8-75d631974635-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2\" (UID: \"57c5d5d3-f260-46e3-9ea8-75d631974635\") " pod="openshift-ingress/payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2"
Nov 03 21:37:17.198348 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.184923    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2"]
Nov 03 21:37:17.319388 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.319351    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c5025c-dockercfg-f89vs\""
Nov 03 21:37:17.319684 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.319667    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c5025c\""
Nov 03 21:37:17.325479 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.324006    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4887\" (UniqueName: \"kubernetes.io/projected/57c5d5d3-f260-46e3-9ea8-75d631974635-kube-api-access-g4887\") pod \"payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2\" (UID: \"57c5d5d3-f260-46e3-9ea8-75d631974635\") " pod="openshift-ingress/payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2"
Nov 03 21:37:17.325479 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.324093    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57c5d5d3-f260-46e3-9ea8-75d631974635-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2\" (UID: \"57c5d5d3-f260-46e3-9ea8-75d631974635\") " pod="openshift-ingress/payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2"
Nov 03 21:37:17.325479 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.325107    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57c5d5d3-f260-46e3-9ea8-75d631974635-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2\" (UID: \"57c5d5d3-f260-46e3-9ea8-75d631974635\") " pod="openshift-ingress/payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2"
Nov 03 21:37:17.377185 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.377155    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9"]
Nov 03 21:37:17.382700 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.382673    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9"
Nov 03 21:37:17.387312 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.384784    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4887\" (UniqueName: \"kubernetes.io/projected/57c5d5d3-f260-46e3-9ea8-75d631974635-kube-api-access-g4887\") pod \"payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2\" (UID: \"57c5d5d3-f260-46e3-9ea8-75d631974635\") " pod="openshift-ingress/payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2"
Nov 03 21:37:17.387312 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.385458    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9c1979a-sd6w9"]
Nov 03 21:37:17.390933 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.390909    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9"]
Nov 03 21:37:17.391206 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.391189    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c1979a-sd6w9"
Nov 03 21:37:17.397284 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.397226    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9c1979a-sd6w9"]
Nov 03 21:37:17.441338 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.431792    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb68k\" (UniqueName: \"kubernetes.io/projected/9d192375-39a1-4b2a-8353-c04b9b4debfd-kube-api-access-fb68k\") pod \"maas-api-revoke-sub-f9c1979a-sd6w9\" (UID: \"9d192375-39a1-4b2a-8353-c04b9b4debfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c1979a-sd6w9"
Nov 03 21:37:17.441338 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.431930    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d192375-39a1-4b2a-8353-c04b9b4debfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9c1979a-sd6w9\" (UID: \"9d192375-39a1-4b2a-8353-c04b9b4debfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c1979a-sd6w9"
Nov 03 21:37:17.441338 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.431962    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d192375-39a1-4b2a-8353-c04b9b4debfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9c1979a-sd6w9\" (UID: \"9d192375-39a1-4b2a-8353-c04b9b4debfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c1979a-sd6w9"
Nov 03 21:37:17.441338 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.432142    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxrj2\" (UniqueName: \"kubernetes.io/projected/2625c6c8-87f6-43df-8607-15c0a2eb32c8-kube-api-access-kxrj2\") pod \"payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9\" (UID: \"2625c6c8-87f6-43df-8607-15c0a2eb32c8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9"
Nov 03 21:37:17.441338 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.432187    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2625c6c8-87f6-43df-8607-15c0a2eb32c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9\" (UID: \"2625c6c8-87f6-43df-8607-15c0a2eb32c8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9"
Nov 03 21:37:17.446303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.446215    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:37:17.446640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.446620    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:37:17.446813 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.446798    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:37:17.446970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.446957    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:37:17.540079 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.540045    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d192375-39a1-4b2a-8353-c04b9b4debfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9c1979a-sd6w9\" (UID: \"9d192375-39a1-4b2a-8353-c04b9b4debfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c1979a-sd6w9"
Nov 03 21:37:17.540079 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.540084    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d192375-39a1-4b2a-8353-c04b9b4debfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9c1979a-sd6w9\" (UID: \"9d192375-39a1-4b2a-8353-c04b9b4debfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c1979a-sd6w9"
Nov 03 21:37:17.540422 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.540206    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxrj2\" (UniqueName: \"kubernetes.io/projected/2625c6c8-87f6-43df-8607-15c0a2eb32c8-kube-api-access-kxrj2\") pod \"payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9\" (UID: \"2625c6c8-87f6-43df-8607-15c0a2eb32c8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9"
Nov 03 21:37:17.540422 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.540227    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2625c6c8-87f6-43df-8607-15c0a2eb32c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9\" (UID: \"2625c6c8-87f6-43df-8607-15c0a2eb32c8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9"
Nov 03 21:37:17.540422 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.540290    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb68k\" (UniqueName: \"kubernetes.io/projected/9d192375-39a1-4b2a-8353-c04b9b4debfd-kube-api-access-fb68k\") pod \"maas-api-revoke-sub-f9c1979a-sd6w9\" (UID: \"9d192375-39a1-4b2a-8353-c04b9b4debfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c1979a-sd6w9"
Nov 03 21:37:17.540697 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.540675    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d192375-39a1-4b2a-8353-c04b9b4debfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9c1979a-sd6w9\" (UID: \"9d192375-39a1-4b2a-8353-c04b9b4debfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c1979a-sd6w9"
Nov 03 21:37:17.541497 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.541470    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2625c6c8-87f6-43df-8607-15c0a2eb32c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9\" (UID: \"2625c6c8-87f6-43df-8607-15c0a2eb32c8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9"
Nov 03 21:37:17.543621 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.543601    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d192375-39a1-4b2a-8353-c04b9b4debfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9c1979a-sd6w9\" (UID: \"9d192375-39a1-4b2a-8353-c04b9b4debfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c1979a-sd6w9"
Nov 03 21:37:17.551367 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.551336    5053 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 03 21:37:17.551465 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.551428    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs" podUID="1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 03 21:37:17.597547 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.597506    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxrj2\" (UniqueName: \"kubernetes.io/projected/2625c6c8-87f6-43df-8607-15c0a2eb32c8-kube-api-access-kxrj2\") pod \"payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9\" (UID: \"2625c6c8-87f6-43df-8607-15c0a2eb32c8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9"
Nov 03 21:37:17.597700 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.597627    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb68k\" (UniqueName: \"kubernetes.io/projected/9d192375-39a1-4b2a-8353-c04b9b4debfd-kube-api-access-fb68k\") pod \"maas-api-revoke-sub-f9c1979a-sd6w9\" (UID: \"9d192375-39a1-4b2a-8353-c04b9b4debfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c1979a-sd6w9"
Nov 03 21:37:17.647303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.646506    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2"
Nov 03 21:37:17.751661 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.751633    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9"
Nov 03 21:37:17.761358 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:17.761189    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c1979a-sd6w9"
Nov 03 21:37:18.069766 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:37:18.069738    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c5d5d3_f260_46e3_9ea8_75d631974635.slice/crio-6f1a7ce02e39b35b1c85a7f41fe0499ed89e6745717de41f9afce97e6b0a05bd WatchSource:0}: Error finding container 6f1a7ce02e39b35b1c85a7f41fe0499ed89e6745717de41f9afce97e6b0a05bd: Status 404 returned error can't find the container with id 6f1a7ce02e39b35b1c85a7f41fe0499ed89e6745717de41f9afce97e6b0a05bd
Nov 03 21:37:18.072475 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:18.072453    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2"]
Nov 03 21:37:18.241544 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:18.241510    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9"]
Nov 03 21:37:18.247357 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:18.247335    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9c1979a-sd6w9"]
Nov 03 21:37:18.342467 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:18.342432    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9" event={"ID":"2625c6c8-87f6-43df-8607-15c0a2eb32c8","Type":"ContainerStarted","Data":"7828bc2379c7750624ffa92bdc695e4ff4b424ec65e78baac82b1dd67f2f10ec"}
Nov 03 21:37:18.344299 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:18.344271    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c1979a-sd6w9" event={"ID":"9d192375-39a1-4b2a-8353-c04b9b4debfd","Type":"ContainerStarted","Data":"1522808acf42aaac89e9867f01340b83627dc5fb3d36f5c0d0e6ab5d6aa18cb3"}
Nov 03 21:37:18.346306 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:18.346207    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2" event={"ID":"57c5d5d3-f260-46e3-9ea8-75d631974635","Type":"ContainerStarted","Data":"88351b4ba63fbcd8a0cd4410b2a7a85b5136d59051e9935f42dfa00fac578714"}
Nov 03 21:37:18.346306 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:18.346278    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2" event={"ID":"57c5d5d3-f260-46e3-9ea8-75d631974635","Type":"ContainerStarted","Data":"6f1a7ce02e39b35b1c85a7f41fe0499ed89e6745717de41f9afce97e6b0a05bd"}
Nov 03 21:37:18.347067 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:18.346536    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2"
Nov 03 21:37:18.461945 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:18.461891    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2" podStartSLOduration=1.461875894 podStartE2EDuration="1.461875894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:18.458511765 +0000 UTC m=+1352.952433640" watchObservedRunningTime="2026-11-03 21:37:18.461875894 +0000 UTC m=+1352.955797770"
Nov 03 21:37:18.551442 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:18.551404    5053 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 03 21:37:18.551650 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:18.551462    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs" podUID="1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 03 21:37:19.356037 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:19.355996    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9" event={"ID":"2625c6c8-87f6-43df-8607-15c0a2eb32c8","Type":"ContainerStarted","Data":"02303d41489a940491c735f79663f32d3e498a8fe47a0ababf8f5e24bc505b6c"}
Nov 03 21:37:19.356492 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:19.356328    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9"
Nov 03 21:37:19.357953 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:19.357928    5053 generic.go:356] "Generic (PLEG): container finished" podID="9d192375-39a1-4b2a-8353-c04b9b4debfd" containerID="e0b071eb3568c72fc00f2226d626ca82304e50a60a5f38e16c41607e2b1bcbce" exitCode=0
Nov 03 21:37:19.358069 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:19.358007    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c1979a-sd6w9" event={"ID":"9d192375-39a1-4b2a-8353-c04b9b4debfd","Type":"ContainerDied","Data":"e0b071eb3568c72fc00f2226d626ca82304e50a60a5f38e16c41607e2b1bcbce"}
Nov 03 21:37:19.483216 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:19.483157    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9" podStartSLOduration=2.483143059 podStartE2EDuration="2.483143059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:19.480757151 +0000 UTC m=+1353.974679026" watchObservedRunningTime="2026-11-03 21:37:19.483143059 +0000 UTC m=+1353.977064933"
Nov 03 21:37:19.551528 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:19.551380    5053 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 03 21:37:19.551528 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:19.551447    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs" podUID="1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 03 21:37:20.551192 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:20.551160    5053 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 03 21:37:20.551596 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:20.551221    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs" podUID="1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 03 21:37:20.556502 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:20.556477    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c1979a-sd6w9"
Nov 03 21:37:20.586373 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:20.586333    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d192375-39a1-4b2a-8353-c04b9b4debfd-kube-api-access-fb68k\" (UniqueName: \"kubernetes.io/projected/9d192375-39a1-4b2a-8353-c04b9b4debfd-kube-api-access-fb68k\") pod \"9d192375-39a1-4b2a-8353-c04b9b4debfd\" (UID: \"9d192375-39a1-4b2a-8353-c04b9b4debfd\") "
Nov 03 21:37:20.586547 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:20.586512    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d192375-39a1-4b2a-8353-c04b9b4debfd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d192375-39a1-4b2a-8353-c04b9b4debfd-maas-api-openshift-ca\") pod \"9d192375-39a1-4b2a-8353-c04b9b4debfd\" (UID: \"9d192375-39a1-4b2a-8353-c04b9b4debfd\") "
Nov 03 21:37:20.586613 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:20.586601    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d192375-39a1-4b2a-8353-c04b9b4debfd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d192375-39a1-4b2a-8353-c04b9b4debfd-maas-api-xks-ca\") pod \"9d192375-39a1-4b2a-8353-c04b9b4debfd\" (UID: \"9d192375-39a1-4b2a-8353-c04b9b4debfd\") "
Nov 03 21:37:20.586894 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:20.586864    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d192375-39a1-4b2a-8353-c04b9b4debfd-maas-api-openshift-ca" pod "9d192375-39a1-4b2a-8353-c04b9b4debfd" (UID: "9d192375-39a1-4b2a-8353-c04b9b4debfd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:37:20.587166 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:20.587148    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d192375-39a1-4b2a-8353-c04b9b4debfd-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:37:20.588634 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:20.588610    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d192375-39a1-4b2a-8353-c04b9b4debfd-kube-api-access-fb68k" pod "9d192375-39a1-4b2a-8353-c04b9b4debfd" (UID: "9d192375-39a1-4b2a-8353-c04b9b4debfd"). InnerVolumeSpecName "kube-api-access-fb68k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:37:20.588711 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:20.588680    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d192375-39a1-4b2a-8353-c04b9b4debfd-maas-api-xks-ca" pod "9d192375-39a1-4b2a-8353-c04b9b4debfd" (UID: "9d192375-39a1-4b2a-8353-c04b9b4debfd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:37:20.688300 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:20.688187    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb68k\" (UniqueName: \"kubernetes.io/projected/9d192375-39a1-4b2a-8353-c04b9b4debfd-kube-api-access-fb68k\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:37:20.688300 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:20.688218    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d192375-39a1-4b2a-8353-c04b9b4debfd-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:37:21.377291 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:21.377224    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c1979a-sd6w9"
Nov 03 21:37:21.377445 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:21.377298    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c1979a-sd6w9" event={"ID":"9d192375-39a1-4b2a-8353-c04b9b4debfd","Type":"ContainerDied","Data":"1522808acf42aaac89e9867f01340b83627dc5fb3d36f5c0d0e6ab5d6aa18cb3"}
Nov 03 21:37:21.377445 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:21.377357    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1522808acf42aaac89e9867f01340b83627dc5fb3d36f5c0d0e6ab5d6aa18cb3"
Nov 03 21:37:21.557760 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:21.557730    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:21.558180 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:21.558104    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:21.558849 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:21.558831    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:37:26.177933 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:26.177903    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr"
Nov 03 21:37:26.178354 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:26.177971    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs"
Nov 03 21:37:27.553872 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.553838    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"]
Nov 03 21:37:27.554652 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.554636    5053 cpu_manager.go:414] "containerMap: removing container" podUID="9d192375-39a1-4b2a-8353-c04b9b4debfd" containerName="revoke-keys"
Nov 03 21:37:27.554714 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.554653    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d192375-39a1-4b2a-8353-c04b9b4debfd" containerName="revoke-keys"
Nov 03 21:37:27.554788 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.554777    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d192375-39a1-4b2a-8353-c04b9b4debfd" containerName="revoke-keys"
Nov 03 21:37:27.559885 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.559865    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:27.611842 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.611810    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 21:37:27.612025 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.611989    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 21:37:27.612025 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.611991    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 21:37:27.612101 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.612037    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-s9fpr\""
Nov 03 21:37:27.675662 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.675632    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"]
Nov 03 21:37:27.694939 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.694906    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8xj6\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:27.695103 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.694952    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8xj6\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:27.695103 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.695063    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7f68c476-0110-4dd1-a322-e7f0e991f56b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8xj6\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:27.695103 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.695089    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnw6g\" (UniqueName: \"kubernetes.io/projected/7f68c476-0110-4dd1-a322-e7f0e991f56b-kube-api-access-dnw6g\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8xj6\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:27.695325 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.695122    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8xj6\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:27.695325 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.695269    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8xj6\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:27.695437 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.695327    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f68c476-0110-4dd1-a322-e7f0e991f56b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8xj6\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:27.796174 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.796141    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7f68c476-0110-4dd1-a322-e7f0e991f56b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8xj6\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:27.796385 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.796186    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnw6g\" (UniqueName: \"kubernetes.io/projected/7f68c476-0110-4dd1-a322-e7f0e991f56b-kube-api-access-dnw6g\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8xj6\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:27.796385 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.796222    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8xj6\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:27.796385 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.796304    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8xj6\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:27.796385 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.796346    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f68c476-0110-4dd1-a322-e7f0e991f56b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8xj6\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:27.796698 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.796665    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8xj6\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:27.796854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.796731    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8xj6\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:27.797230 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.797205    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8xj6\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:27.798825 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.798796    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f68c476-0110-4dd1-a322-e7f0e991f56b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8xj6\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:27.798899 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.798828    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7f68c476-0110-4dd1-a322-e7f0e991f56b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8xj6\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:27.799023 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.799005    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8xj6\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:27.799170 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.799154    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8xj6\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:27.853978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.853907    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8xj6\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:27.854100 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.854054    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnw6g\" (UniqueName: \"kubernetes.io/projected/7f68c476-0110-4dd1-a322-e7f0e991f56b-kube-api-access-dnw6g\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8xj6\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:27.873944 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:27.873916    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:28.313957 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:37:28.313924    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f68c476_0110_4dd1_a322_e7f0e991f56b.slice/crio-b96a28d0e7eb54b60fac9f08458806e68ef66a817cd116d0f11e9007a56d41d0 WatchSource:0}: Error finding container b96a28d0e7eb54b60fac9f08458806e68ef66a817cd116d0f11e9007a56d41d0: Status 404 returned error can't find the container with id b96a28d0e7eb54b60fac9f08458806e68ef66a817cd116d0f11e9007a56d41d0
Nov 03 21:37:28.314367 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:28.314345    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"]
Nov 03 21:37:28.316446 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:28.316429    5053 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:37:28.444594 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:28.444557    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6" event={"ID":"7f68c476-0110-4dd1-a322-e7f0e991f56b","Type":"ContainerStarted","Data":"b96a28d0e7eb54b60fac9f08458806e68ef66a817cd116d0f11e9007a56d41d0"}
Nov 03 21:37:29.361619 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:29.361590    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2"
Nov 03 21:37:30.371701 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:30.371672    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9"
Nov 03 21:37:32.166543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:32.166510    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:37:32.166848 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:32.166572    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:37:32.499775 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:32.499738    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6" event={"ID":"7f68c476-0110-4dd1-a322-e7f0e991f56b","Type":"ContainerStarted","Data":"2413071844b82bf1f0a0c36fc99c76cbfd94778e0ca1cdddfc3c042e70bf3117"}
Nov 03 21:37:32.499977 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:32.499794    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:32.666141 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:32.666087    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6" podStartSLOduration=1.817371551 podStartE2EDuration="5.666069153s" totalImagesPullingTime="3.848697602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:37:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:32.664778448 +0000 UTC m=+1367.158700323" watchObservedRunningTime="2026-11-03 21:37:32.666069153 +0000 UTC m=+1367.159991027"
Nov 03 21:37:33.517376 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:33.517341    5053 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-v8xj6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:37:33.517914 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:33.517880    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6" podUID="7f68c476-0110-4dd1-a322-e7f0e991f56b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:37:36.512218 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:36.512186    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:37:41.081461 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:37:41.081408    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:37:46.216296 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:37:46.216266    5053 scope.go:122] "RemoveContainer" containerID="5233a4f6be8f6022c92e12ff14e181f14cc9661dcfd371def2db5cf95d6a2148"
Nov 03 21:38:10.930787 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:38:10.930743    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:38:11.047605 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:11.047562    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd75a1ed-59df5"]
Nov 03 21:38:11.097617 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:11.097575    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd75a1ed-59df5"]
Nov 03 21:38:11.669808 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:11.669734    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f572ca4a-00b0-43b2-9681-91680b27a69b" path="/var/lib/kubelet/pods/f572ca4a-00b0-43b2-9681-91680b27a69b/volumes"
Nov 03 21:38:12.108105 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:12.108072    5053 generic.go:356] "Generic (PLEG): container finished" podID="7b508d7b-1822-47fa-9565-61da014df9c3" containerID="8b1315fa3b93f0a626aab55322e8fe8ac790bd1e711f4cbdfca21488ad0e7e3d" exitCode=0
Nov 03 21:38:12.108668 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:12.108146    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9" event={"ID":"7b508d7b-1822-47fa-9565-61da014df9c3","Type":"ContainerDied","Data":"8b1315fa3b93f0a626aab55322e8fe8ac790bd1e711f4cbdfca21488ad0e7e3d"}
Nov 03 21:38:12.108751 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:12.108693    5053 scope.go:122] "RemoveContainer" containerID="8b1315fa3b93f0a626aab55322e8fe8ac790bd1e711f4cbdfca21488ad0e7e3d"
Nov 03 21:38:12.110167 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:12.110134    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:38:12.110310 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:12.110218    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:38:12.110381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:12.110315    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:38:13.139992 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:13.139954    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9" event={"ID":"7b508d7b-1822-47fa-9565-61da014df9c3","Type":"ContainerStarted","Data":"cb84fc8f0c298d92912fa7fa21ac1bd7ddf56b9d9115f2a684d8fce7c65c0d3d"}
Nov 03 21:38:13.886315 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:13.886277    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:38:13.886535 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:13.886515    5053 patch_prober.go:29] interesting pod/e2e-api-a-29472d-openshift-default-55bd446969-f54r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 03 21:38:13.886585 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:13.886561    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9" podUID="7b508d7b-1822-47fa-9565-61da014df9c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 03 21:38:14.887117 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:14.887083    5053 patch_prober.go:29] interesting pod/e2e-api-a-29472d-openshift-default-55bd446969-f54r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 03 21:38:14.887637 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:14.887156    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9" podUID="7b508d7b-1822-47fa-9565-61da014df9c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 03 21:38:15.886404 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:15.886370    5053 patch_prober.go:29] interesting pod/e2e-api-a-29472d-openshift-default-55bd446969-f54r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 03 21:38:15.886591 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:15.886437    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9" podUID="7b508d7b-1822-47fa-9565-61da014df9c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 03 21:38:16.886772 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:16.886742    5053 patch_prober.go:29] interesting pod/e2e-api-a-29472d-openshift-default-55bd446969-f54r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 03 21:38:16.887255 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:16.886802    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9" podUID="7b508d7b-1822-47fa-9565-61da014df9c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 03 21:38:17.886684 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:17.886649    5053 patch_prober.go:29] interesting pod/e2e-api-a-29472d-openshift-default-55bd446969-f54r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 03 21:38:17.886955 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:17.886705    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9" podUID="7b508d7b-1822-47fa-9565-61da014df9c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 03 21:38:18.891641 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:18.891608    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:38:18.892137 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:18.892005    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:38:18.892638 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:18.892619    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:38:26.048732 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:26.048691    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2d3ea98-zcqh9"]
Nov 03 21:38:26.053503 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:26.053473    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2d3ea98-zcqh9"]
Nov 03 21:38:27.670834 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:27.670794    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f120939-31ab-4bc2-8773-42fbd6268939" path="/var/lib/kubelet/pods/2f120939-31ab-4bc2-8773-42fbd6268939/volumes"
Nov 03 21:38:38.050306 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:38.050264    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4466cc7-rbvjl"]
Nov 03 21:38:38.096900 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:38.096867    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4466cc7-rbvjl"]
Nov 03 21:38:39.668561 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:39.668524    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad574db0-89e0-4a72-84bf-bbd62702b536" path="/var/lib/kubelet/pods/ad574db0-89e0-4a72-84bf-bbd62702b536/volumes"
Nov 03 21:38:40.098536 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:40.097310    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79b925eb-lpm74"]
Nov 03 21:38:40.153825 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:40.153789    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79b925eb-lpm74"]
Nov 03 21:38:40.931189 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:38:40.931149    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:38:41.669469 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:41.669427    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da076dbe-da94-4097-94e7-f133355e277d" path="/var/lib/kubelet/pods/da076dbe-da94-4097-94e7-f133355e277d/volumes"
Nov 03 21:38:46.247136 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:46.247097    5053 scope.go:122] "RemoveContainer" containerID="8ac5b01b14d625cec79d9287ecf067e5af8f04a1dd00d706ab67fe200a10da05"
Nov 03 21:38:46.267110 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:46.267084    5053 scope.go:122] "RemoveContainer" containerID="fdc801a59da4723f34101bb058bab315ff0a7c4c0c0815d1643ae23ef3482e02"
Nov 03 21:38:49.048152 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:49.048118    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d6a96e0-ndbbl"]
Nov 03 21:38:49.097537 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:49.097503    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d6a96e0-ndbbl"]
Nov 03 21:38:49.668304 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:49.668222    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e72ac212-56b7-4d76-a32f-6ee939a927b5" path="/var/lib/kubelet/pods/e72ac212-56b7-4d76-a32f-6ee939a927b5/volumes"
Nov 03 21:38:58.047016 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:58.046980    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-142a67da-vh2r8"]
Nov 03 21:38:58.095786 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:58.095748    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-142a67da-vh2r8"]
Nov 03 21:38:59.049678 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.049628    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"]
Nov 03 21:38:59.068298 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.056821    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.136466 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.136430    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"]
Nov 03 21:38:59.137719 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.137689    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-podinfo\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.138352 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.138329    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-credential-socket\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.138516 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.138500    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istiod-ca-cert\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.138617 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.138605    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbtbz\" (UniqueName: \"kubernetes.io/projected/83a53cc7-96e4-49db-a8ac-2d2de9e84188-kube-api-access-kbtbz\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.138853 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.138835    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-envoy\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.139034 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.139020    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-token\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.139157 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.139145    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-data\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.139344 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.139329    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-workload-certs\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.139463 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.139449    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-workload-socket\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.171366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.163070    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-fff189-openshift-default-dockercfg-z8znp\""
Nov 03 21:38:59.250226 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.250134    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-podinfo\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.250529 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.250508    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-credential-socket\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.250653 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.250641    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istiod-ca-cert\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.250759 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.250747    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbtbz\" (UniqueName: \"kubernetes.io/projected/83a53cc7-96e4-49db-a8ac-2d2de9e84188-kube-api-access-kbtbz\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.250970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.250958    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-envoy\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.251123 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.251111    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-token\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.251260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.251225    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-data\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.251389 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.251372    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-workload-certs\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.251572 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.251554    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-workload-socket\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.252226 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.252204    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-workload-socket\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.260385 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.260347    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-credential-socket\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.264784 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.263848    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istiod-ca-cert\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.264784 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.264353    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-data\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.264784 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.264747    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-workload-certs\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.267983 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.267942    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-envoy\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.314404 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.291931    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-podinfo\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.385868 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.385790    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbtbz\" (UniqueName: \"kubernetes.io/projected/83a53cc7-96e4-49db-a8ac-2d2de9e84188-kube-api-access-kbtbz\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.406372 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.402829    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-token\") pod \"e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") " pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.439259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.438588    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:38:59.670930 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.670848    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26ce17c8-b884-4902-b42a-b83005710f38" path="/var/lib/kubelet/pods/26ce17c8-b884-4902-b42a-b83005710f38/volumes"
Nov 03 21:38:59.921148 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:38:59.915925    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a53cc7_96e4_49db_a8ac_2d2de9e84188.slice/crio-7c46de45d621afbb5b5a6d0b8ba69a41609d0457d7d736cb54bb4828201a4e52 WatchSource:0}: Error finding container 7c46de45d621afbb5b5a6d0b8ba69a41609d0457d7d736cb54bb4828201a4e52: Status 404 returned error can't find the container with id 7c46de45d621afbb5b5a6d0b8ba69a41609d0457d7d736cb54bb4828201a4e52
Nov 03 21:38:59.921148 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.916030    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"]
Nov 03 21:38:59.924069 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.924033    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:38:59.924168 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.924105    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:38:59.924168 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:38:59.924135    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:39:00.809023 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:00.808985    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq" event={"ID":"83a53cc7-96e4-49db-a8ac-2d2de9e84188","Type":"ContainerStarted","Data":"16be4c649a9cbd5baa803ba185f0dcc51e312bf4e55767a02b20d64a20d72882"}
Nov 03 21:39:00.809511 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:00.809044    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq" event={"ID":"83a53cc7-96e4-49db-a8ac-2d2de9e84188","Type":"ContainerStarted","Data":"7c46de45d621afbb5b5a6d0b8ba69a41609d0457d7d736cb54bb4828201a4e52"}
Nov 03 21:39:00.928960 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:00.928902    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq" podStartSLOduration=2.928886303 podStartE2EDuration="2.928886303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:00.925288145 +0000 UTC m=+1455.419210019" watchObservedRunningTime="2026-11-03 21:39:00.928886303 +0000 UTC m=+1455.422808242"
Nov 03 21:39:01.010153 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:01.010119    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-29s5p"]
Nov 03 21:39:01.019370 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:01.019340    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-29s5p"
Nov 03 21:39:01.069789 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:01.069698    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-29s5p"]
Nov 03 21:39:01.071685 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:01.071661    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:39:01.071880 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:01.071703    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:39:01.099661 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:01.099611    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgfv9\" (UniqueName: \"kubernetes.io/projected/ba223252-af37-4d5f-b5c0-ccb1e1f91c05-kube-api-access-pgfv9\") pod \"maas-api-revoke-keys-e2e-mt-c1662df5-29s5p\" (UID: \"ba223252-af37-4d5f-b5c0-ccb1e1f91c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-29s5p"
Nov 03 21:39:01.099890 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:01.099829    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba223252-af37-4d5f-b5c0-ccb1e1f91c05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1662df5-29s5p\" (UID: \"ba223252-af37-4d5f-b5c0-ccb1e1f91c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-29s5p"
Nov 03 21:39:01.099958 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:01.099940    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba223252-af37-4d5f-b5c0-ccb1e1f91c05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1662df5-29s5p\" (UID: \"ba223252-af37-4d5f-b5c0-ccb1e1f91c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-29s5p"
Nov 03 21:39:01.132558 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:01.132489    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:39:01.132975 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:01.132945    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:39:01.201214 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:01.201176    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba223252-af37-4d5f-b5c0-ccb1e1f91c05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1662df5-29s5p\" (UID: \"ba223252-af37-4d5f-b5c0-ccb1e1f91c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-29s5p"
Nov 03 21:39:01.201449 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:01.201346    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba223252-af37-4d5f-b5c0-ccb1e1f91c05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1662df5-29s5p\" (UID: \"ba223252-af37-4d5f-b5c0-ccb1e1f91c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-29s5p"
Nov 03 21:39:01.201521 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:01.201498    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgfv9\" (UniqueName: \"kubernetes.io/projected/ba223252-af37-4d5f-b5c0-ccb1e1f91c05-kube-api-access-pgfv9\") pod \"maas-api-revoke-keys-e2e-mt-c1662df5-29s5p\" (UID: \"ba223252-af37-4d5f-b5c0-ccb1e1f91c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-29s5p"
Nov 03 21:39:01.202102 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:01.202049    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba223252-af37-4d5f-b5c0-ccb1e1f91c05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1662df5-29s5p\" (UID: \"ba223252-af37-4d5f-b5c0-ccb1e1f91c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-29s5p"
Nov 03 21:39:01.203997 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:01.203974    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba223252-af37-4d5f-b5c0-ccb1e1f91c05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1662df5-29s5p\" (UID: \"ba223252-af37-4d5f-b5c0-ccb1e1f91c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-29s5p"
Nov 03 21:39:01.258402 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:01.258367    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgfv9\" (UniqueName: \"kubernetes.io/projected/ba223252-af37-4d5f-b5c0-ccb1e1f91c05-kube-api-access-pgfv9\") pod \"maas-api-revoke-keys-e2e-mt-c1662df5-29s5p\" (UID: \"ba223252-af37-4d5f-b5c0-ccb1e1f91c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-29s5p"
Nov 03 21:39:01.345593 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:01.345498    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-29s5p"
Nov 03 21:39:01.441037 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:01.439386    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:39:01.446727 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:01.446694    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:39:01.724908 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:39:01.724864    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba223252_af37_4d5f_b5c0_ccb1e1f91c05.slice/crio-00d89c558fcee8219beb9678eda8bbce87c9a68443ee513dcf4312cec851833a WatchSource:0}: Error finding container 00d89c558fcee8219beb9678eda8bbce87c9a68443ee513dcf4312cec851833a: Status 404 returned error can't find the container with id 00d89c558fcee8219beb9678eda8bbce87c9a68443ee513dcf4312cec851833a
Nov 03 21:39:01.726814 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:01.726784    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-29s5p"]
Nov 03 21:39:01.822702 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:01.822658    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-29s5p" event={"ID":"ba223252-af37-4d5f-b5c0-ccb1e1f91c05","Type":"ContainerStarted","Data":"00d89c558fcee8219beb9678eda8bbce87c9a68443ee513dcf4312cec851833a"}
Nov 03 21:39:01.823121 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:01.822978    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:39:01.824297 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:01.824274    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:39:02.835875 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:02.835837    5053 generic.go:356] "Generic (PLEG): container finished" podID="ba223252-af37-4d5f-b5c0-ccb1e1f91c05" containerID="39e7503a67875ad2c1c7b846a02997c55e5554593d4741c2073a54aa50abecd7" exitCode=0
Nov 03 21:39:02.836379 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:02.835915    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-29s5p" event={"ID":"ba223252-af37-4d5f-b5c0-ccb1e1f91c05","Type":"ContainerDied","Data":"39e7503a67875ad2c1c7b846a02997c55e5554593d4741c2073a54aa50abecd7"}
Nov 03 21:39:04.060671 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:04.060637    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-29s5p"
Nov 03 21:39:04.156070 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:04.155979    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba223252-af37-4d5f-b5c0-ccb1e1f91c05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba223252-af37-4d5f-b5c0-ccb1e1f91c05-maas-api-openshift-ca\") pod \"ba223252-af37-4d5f-b5c0-ccb1e1f91c05\" (UID: \"ba223252-af37-4d5f-b5c0-ccb1e1f91c05\") "
Nov 03 21:39:04.156070 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:04.156044    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba223252-af37-4d5f-b5c0-ccb1e1f91c05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba223252-af37-4d5f-b5c0-ccb1e1f91c05-maas-api-xks-ca\") pod \"ba223252-af37-4d5f-b5c0-ccb1e1f91c05\" (UID: \"ba223252-af37-4d5f-b5c0-ccb1e1f91c05\") "
Nov 03 21:39:04.156379 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:04.156214    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba223252-af37-4d5f-b5c0-ccb1e1f91c05-kube-api-access-pgfv9\" (UniqueName: \"kubernetes.io/projected/ba223252-af37-4d5f-b5c0-ccb1e1f91c05-kube-api-access-pgfv9\") pod \"ba223252-af37-4d5f-b5c0-ccb1e1f91c05\" (UID: \"ba223252-af37-4d5f-b5c0-ccb1e1f91c05\") "
Nov 03 21:39:04.156491 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:04.156455    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba223252-af37-4d5f-b5c0-ccb1e1f91c05-maas-api-openshift-ca" pod "ba223252-af37-4d5f-b5c0-ccb1e1f91c05" (UID: "ba223252-af37-4d5f-b5c0-ccb1e1f91c05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:39:04.156890 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:04.156833    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba223252-af37-4d5f-b5c0-ccb1e1f91c05-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:39:04.158486 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:04.158464    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba223252-af37-4d5f-b5c0-ccb1e1f91c05-maas-api-xks-ca" pod "ba223252-af37-4d5f-b5c0-ccb1e1f91c05" (UID: "ba223252-af37-4d5f-b5c0-ccb1e1f91c05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:39:04.158722 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:04.158706    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba223252-af37-4d5f-b5c0-ccb1e1f91c05-kube-api-access-pgfv9" pod "ba223252-af37-4d5f-b5c0-ccb1e1f91c05" (UID: "ba223252-af37-4d5f-b5c0-ccb1e1f91c05"). InnerVolumeSpecName "kube-api-access-pgfv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:39:04.257974 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:04.257939    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba223252-af37-4d5f-b5c0-ccb1e1f91c05-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:39:04.257974 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:04.257970    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgfv9\" (UniqueName: \"kubernetes.io/projected/ba223252-af37-4d5f-b5c0-ccb1e1f91c05-kube-api-access-pgfv9\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:39:04.857531 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:04.857499    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-29s5p"
Nov 03 21:39:04.857705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:04.857531    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-29s5p" event={"ID":"ba223252-af37-4d5f-b5c0-ccb1e1f91c05","Type":"ContainerDied","Data":"00d89c558fcee8219beb9678eda8bbce87c9a68443ee513dcf4312cec851833a"}
Nov 03 21:39:04.857705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:04.857576    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00d89c558fcee8219beb9678eda8bbce87c9a68443ee513dcf4312cec851833a"
Nov 03 21:39:05.054192 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:05.054160    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c39782a-rvqsn"]
Nov 03 21:39:05.061683 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:05.061641    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c39782a-rvqsn"]
Nov 03 21:39:05.669760 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:05.669718    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a55e0e3c-bdb8-4ed0-9756-99a0f952584d" path="/var/lib/kubelet/pods/a55e0e3c-bdb8-4ed0-9756-99a0f952584d/volumes"
Nov 03 21:39:10.930218 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:39:10.930178    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:39:11.047348 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:11.047309    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ea73bd-zvfss"]
Nov 03 21:39:11.096559 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:11.096524    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ea73bd-zvfss"]
Nov 03 21:39:11.669626 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:11.669591    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1809882-daeb-48eb-82ac-76797b259c1c" path="/var/lib/kubelet/pods/e1809882-daeb-48eb-82ac-76797b259c1c/volumes"
Nov 03 21:39:12.033081 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:12.033052    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de3a73dd-bz6x9"]
Nov 03 21:39:12.033838 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:12.033823    5053 cpu_manager.go:414] "containerMap: removing container" podUID="ba223252-af37-4d5f-b5c0-ccb1e1f91c05" containerName="revoke-keys"
Nov 03 21:39:12.033879 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:12.033839    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba223252-af37-4d5f-b5c0-ccb1e1f91c05" containerName="revoke-keys"
Nov 03 21:39:12.033994 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:12.033985    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba223252-af37-4d5f-b5c0-ccb1e1f91c05" containerName="revoke-keys"
Nov 03 21:39:12.037359 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:12.037340    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3a73dd-bz6x9"
Nov 03 21:39:12.097588 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:12.097546    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de3a73dd-bz6x9\" (UID: \"1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3a73dd-bz6x9"
Nov 03 21:39:12.097793 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:12.097774    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de3a73dd-bz6x9\" (UID: \"1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3a73dd-bz6x9"
Nov 03 21:39:12.098052 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:12.098029    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl5t7\" (UniqueName: \"kubernetes.io/projected/1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f-kube-api-access-sl5t7\") pod \"maas-api-revoke-sub-de3a73dd-bz6x9\" (UID: \"1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3a73dd-bz6x9"
Nov 03 21:39:12.098504 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:12.098476    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de3a73dd-bz6x9"]
Nov 03 21:39:12.139781 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:12.139745    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:39:12.139945 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:12.139789    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:39:12.139945 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:12.139837    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:39:12.139945 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:12.139861    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:39:12.199170 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:12.199140    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de3a73dd-bz6x9\" (UID: \"1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3a73dd-bz6x9"
Nov 03 21:39:12.199365 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:12.199311    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl5t7\" (UniqueName: \"kubernetes.io/projected/1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f-kube-api-access-sl5t7\") pod \"maas-api-revoke-sub-de3a73dd-bz6x9\" (UID: \"1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3a73dd-bz6x9"
Nov 03 21:39:12.199423 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:12.199410    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de3a73dd-bz6x9\" (UID: \"1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3a73dd-bz6x9"
Nov 03 21:39:12.199798 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:12.199775    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de3a73dd-bz6x9\" (UID: \"1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3a73dd-bz6x9"
Nov 03 21:39:12.202011 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:12.201985    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de3a73dd-bz6x9\" (UID: \"1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3a73dd-bz6x9"
Nov 03 21:39:12.256398 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:12.256373    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl5t7\" (UniqueName: \"kubernetes.io/projected/1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f-kube-api-access-sl5t7\") pod \"maas-api-revoke-sub-de3a73dd-bz6x9\" (UID: \"1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3a73dd-bz6x9"
Nov 03 21:39:12.350865 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:12.350785    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3a73dd-bz6x9"
Nov 03 21:39:12.744963 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:12.744933    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de3a73dd-bz6x9"]
Nov 03 21:39:12.884843 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:12.884806    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm"]
Nov 03 21:39:12.935206 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:12.935167    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3a73dd-bz6x9" event={"ID":"1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f","Type":"ContainerStarted","Data":"243ba7e789f187540e0ed526f72b29d471d1c8522fd33a47883ed05d251106ad"}
Nov 03 21:39:13.005421 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:13.005340    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg"]
Nov 03 21:39:13.945103 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:13.945066    5053 generic.go:356] "Generic (PLEG): container finished" podID="1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f" containerID="75bef1aaf411c95c4a183a16e79c7ec58110c4f066a7214e292a136bb239bedb" exitCode=0
Nov 03 21:39:13.945507 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:13.945145    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3a73dd-bz6x9" event={"ID":"1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f","Type":"ContainerDied","Data":"75bef1aaf411c95c4a183a16e79c7ec58110c4f066a7214e292a136bb239bedb"}
Nov 03 21:39:15.168898 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:15.168877    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3a73dd-bz6x9"
Nov 03 21:39:15.278029 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:15.277943    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f-maas-api-xks-ca\") pod \"1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f\" (UID: \"1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f\") "
Nov 03 21:39:15.278206 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:15.278159    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f-maas-api-openshift-ca\") pod \"1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f\" (UID: \"1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f\") "
Nov 03 21:39:15.278298 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:15.278208    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f-kube-api-access-sl5t7\" (UniqueName: \"kubernetes.io/projected/1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f-kube-api-access-sl5t7\") pod \"1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f\" (UID: \"1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f\") "
Nov 03 21:39:15.278516 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:15.278489    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f-maas-api-openshift-ca" pod "1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f" (UID: "1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:39:15.278909 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:15.278888    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:39:15.280398 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:15.280374    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f-maas-api-xks-ca" pod "1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f" (UID: "1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:39:15.280474 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:15.280430    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f-kube-api-access-sl5t7" pod "1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f" (UID: "1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f"). InnerVolumeSpecName "kube-api-access-sl5t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:39:15.380106 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:15.380072    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl5t7\" (UniqueName: \"kubernetes.io/projected/1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f-kube-api-access-sl5t7\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:39:15.380106 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:15.380103    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:39:15.963363 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:15.963330    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3a73dd-bz6x9"
Nov 03 21:39:15.963547 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:15.963367    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3a73dd-bz6x9" event={"ID":"1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f","Type":"ContainerDied","Data":"243ba7e789f187540e0ed526f72b29d471d1c8522fd33a47883ed05d251106ad"}
Nov 03 21:39:15.963547 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:15.963420    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="243ba7e789f187540e0ed526f72b29d471d1c8522fd33a47883ed05d251106ad"
Nov 03 21:39:17.885596 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:17.885532    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm" podUID="0ead33cd-8e19-422f-bef3-4e069ba97eda" containerName="payload-processing" containerID="cri-o://f8254eebaa7afc825f982670652adaf8ad2e403db598a995ed376a541ba30427" gracePeriod=25
Nov 03 21:39:18.006586 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:18.006538    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg" podUID="5553e95b-04c9-4716-a5c6-8f5cdf96bbd2" containerName="payload-pre-processing" containerID="cri-o://b77c5e59ae830c46d2a13719615f34c92196610eb4e1700253ec7f85a9b5f26f" gracePeriod=25
Nov 03 21:39:18.194011 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:18.193986    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm"
Nov 03 21:39:18.236940 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:18.233776    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ead33cd-8e19-422f-bef3-4e069ba97eda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ead33cd-8e19-422f-bef3-4e069ba97eda-plugins-config-volume\") pod \"0ead33cd-8e19-422f-bef3-4e069ba97eda\" (UID: \"0ead33cd-8e19-422f-bef3-4e069ba97eda\") "
Nov 03 21:39:18.236940 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:18.234361    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ead33cd-8e19-422f-bef3-4e069ba97eda-plugins-config-volume" pod "0ead33cd-8e19-422f-bef3-4e069ba97eda" (UID: "0ead33cd-8e19-422f-bef3-4e069ba97eda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:39:18.236940 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:18.234522    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ead33cd-8e19-422f-bef3-4e069ba97eda-kube-api-access-84tpq\" (UniqueName: \"kubernetes.io/projected/0ead33cd-8e19-422f-bef3-4e069ba97eda-kube-api-access-84tpq\") pod \"0ead33cd-8e19-422f-bef3-4e069ba97eda\" (UID: \"0ead33cd-8e19-422f-bef3-4e069ba97eda\") "
Nov 03 21:39:18.236940 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:18.235645    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ead33cd-8e19-422f-bef3-4e069ba97eda-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:39:18.253515 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:18.253426    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ead33cd-8e19-422f-bef3-4e069ba97eda-kube-api-access-84tpq" pod "0ead33cd-8e19-422f-bef3-4e069ba97eda" (UID: "0ead33cd-8e19-422f-bef3-4e069ba97eda"). InnerVolumeSpecName "kube-api-access-84tpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:39:18.298756 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:18.298731    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg"
Nov 03 21:39:18.337284 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:18.337218    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5553e95b-04c9-4716-a5c6-8f5cdf96bbd2-kube-api-access-f7hwf\" (UniqueName: \"kubernetes.io/projected/5553e95b-04c9-4716-a5c6-8f5cdf96bbd2-kube-api-access-f7hwf\") pod \"5553e95b-04c9-4716-a5c6-8f5cdf96bbd2\" (UID: \"5553e95b-04c9-4716-a5c6-8f5cdf96bbd2\") "
Nov 03 21:39:18.337602 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:18.337576    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5553e95b-04c9-4716-a5c6-8f5cdf96bbd2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5553e95b-04c9-4716-a5c6-8f5cdf96bbd2-plugins-config-volume\") pod \"5553e95b-04c9-4716-a5c6-8f5cdf96bbd2\" (UID: \"5553e95b-04c9-4716-a5c6-8f5cdf96bbd2\") "
Nov 03 21:39:18.338294 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:18.338273    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84tpq\" (UniqueName: \"kubernetes.io/projected/0ead33cd-8e19-422f-bef3-4e069ba97eda-kube-api-access-84tpq\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:39:18.338651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:18.338629    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5553e95b-04c9-4716-a5c6-8f5cdf96bbd2-plugins-config-volume" pod "5553e95b-04c9-4716-a5c6-8f5cdf96bbd2" (UID: "5553e95b-04c9-4716-a5c6-8f5cdf96bbd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:39:18.340347 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:18.340323    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5553e95b-04c9-4716-a5c6-8f5cdf96bbd2-kube-api-access-f7hwf" pod "5553e95b-04c9-4716-a5c6-8f5cdf96bbd2" (UID: "5553e95b-04c9-4716-a5c6-8f5cdf96bbd2"). InnerVolumeSpecName "kube-api-access-f7hwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:39:18.440304 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:18.440192    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7hwf\" (UniqueName: \"kubernetes.io/projected/5553e95b-04c9-4716-a5c6-8f5cdf96bbd2-kube-api-access-f7hwf\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:39:18.440516 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:18.440497    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5553e95b-04c9-4716-a5c6-8f5cdf96bbd2-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:39:18.990128 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:18.990091    5053 generic.go:356] "Generic (PLEG): container finished" podID="5553e95b-04c9-4716-a5c6-8f5cdf96bbd2" containerID="b77c5e59ae830c46d2a13719615f34c92196610eb4e1700253ec7f85a9b5f26f" exitCode=0
Nov 03 21:39:18.990597 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:18.990135    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg"
Nov 03 21:39:18.990597 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:18.990174    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg" event={"ID":"5553e95b-04c9-4716-a5c6-8f5cdf96bbd2","Type":"ContainerDied","Data":"b77c5e59ae830c46d2a13719615f34c92196610eb4e1700253ec7f85a9b5f26f"}
Nov 03 21:39:18.990597 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:18.990227    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg" event={"ID":"5553e95b-04c9-4716-a5c6-8f5cdf96bbd2","Type":"ContainerDied","Data":"43d7ed6b6bb4b48c9a64f3c27f57dcff220e1d8d94e44643b679455243122029"}
Nov 03 21:39:18.990597 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:18.990273    5053 scope.go:122] "RemoveContainer" containerID="b77c5e59ae830c46d2a13719615f34c92196610eb4e1700253ec7f85a9b5f26f"
Nov 03 21:39:18.991761 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:18.991739    5053 generic.go:356] "Generic (PLEG): container finished" podID="0ead33cd-8e19-422f-bef3-4e069ba97eda" containerID="f8254eebaa7afc825f982670652adaf8ad2e403db598a995ed376a541ba30427" exitCode=0
Nov 03 21:39:18.991859 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:18.991787    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm"
Nov 03 21:39:18.991859 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:18.991819    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm" event={"ID":"0ead33cd-8e19-422f-bef3-4e069ba97eda","Type":"ContainerDied","Data":"f8254eebaa7afc825f982670652adaf8ad2e403db598a995ed376a541ba30427"}
Nov 03 21:39:18.991941 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:18.991857    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm" event={"ID":"0ead33cd-8e19-422f-bef3-4e069ba97eda","Type":"ContainerDied","Data":"a9ae6a5c2aaf25cd5f3ec95f02f582d35cd14ddc0b88a9990dfea60d3f1d05e3"}
Nov 03 21:39:19.007495 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:19.007470    5053 scope.go:122] "RemoveContainer" containerID="b77c5e59ae830c46d2a13719615f34c92196610eb4e1700253ec7f85a9b5f26f"
Nov 03 21:39:19.007929 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:39:19.007897    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b77c5e59ae830c46d2a13719615f34c92196610eb4e1700253ec7f85a9b5f26f\": container with ID starting with b77c5e59ae830c46d2a13719615f34c92196610eb4e1700253ec7f85a9b5f26f not found: ID does not exist" containerID="b77c5e59ae830c46d2a13719615f34c92196610eb4e1700253ec7f85a9b5f26f"
Nov 03 21:39:19.008027 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:19.007937    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b77c5e59ae830c46d2a13719615f34c92196610eb4e1700253ec7f85a9b5f26f"} err="failed to get container status \"b77c5e59ae830c46d2a13719615f34c92196610eb4e1700253ec7f85a9b5f26f\": rpc error: code = NotFound desc = could not find container \"b77c5e59ae830c46d2a13719615f34c92196610eb4e1700253ec7f85a9b5f26f\": container with ID starting with b77c5e59ae830c46d2a13719615f34c92196610eb4e1700253ec7f85a9b5f26f not found: ID does not exist"
Nov 03 21:39:19.008027 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:19.007962    5053 scope.go:122] "RemoveContainer" containerID="f8254eebaa7afc825f982670652adaf8ad2e403db598a995ed376a541ba30427"
Nov 03 21:39:19.020664 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:19.020638    5053 scope.go:122] "RemoveContainer" containerID="f8254eebaa7afc825f982670652adaf8ad2e403db598a995ed376a541ba30427"
Nov 03 21:39:19.020988 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:39:19.020963    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8254eebaa7afc825f982670652adaf8ad2e403db598a995ed376a541ba30427\": container with ID starting with f8254eebaa7afc825f982670652adaf8ad2e403db598a995ed376a541ba30427 not found: ID does not exist" containerID="f8254eebaa7afc825f982670652adaf8ad2e403db598a995ed376a541ba30427"
Nov 03 21:39:19.021076 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:19.020999    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8254eebaa7afc825f982670652adaf8ad2e403db598a995ed376a541ba30427"} err="failed to get container status \"f8254eebaa7afc825f982670652adaf8ad2e403db598a995ed376a541ba30427\": rpc error: code = NotFound desc = could not find container \"f8254eebaa7afc825f982670652adaf8ad2e403db598a995ed376a541ba30427\": container with ID starting with f8254eebaa7afc825f982670652adaf8ad2e403db598a995ed376a541ba30427 not found: ID does not exist"
Nov 03 21:39:19.155153 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:19.155066    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg"]
Nov 03 21:39:19.204645 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:19.204607    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-6zptg"]
Nov 03 21:39:19.267308 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:19.267277    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm"]
Nov 03 21:39:19.271572 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:19.271545    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-9fnhm"]
Nov 03 21:39:19.668515 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:19.668485    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ead33cd-8e19-422f-bef3-4e069ba97eda" path="/var/lib/kubelet/pods/0ead33cd-8e19-422f-bef3-4e069ba97eda/volumes"
Nov 03 21:39:19.668976 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:19.668961    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5553e95b-04c9-4716-a5c6-8f5cdf96bbd2" path="/var/lib/kubelet/pods/5553e95b-04c9-4716-a5c6-8f5cdf96bbd2/volumes"
Nov 03 21:39:22.048874 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:22.048838    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e52642e-l6wlq"]
Nov 03 21:39:22.103262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:22.101406    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e52642e-l6wlq"]
Nov 03 21:39:23.668346 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:23.668314    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4f71ae5-f966-4a7b-9181-6151aa57f43f" path="/var/lib/kubelet/pods/c4f71ae5-f966-4a7b-9181-6151aa57f43f/volumes"
Nov 03 21:39:29.049707 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:29.049621    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb40886b-64qgl"]
Nov 03 21:39:29.098807 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:29.098770    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb40886b-64qgl"]
Nov 03 21:39:29.670815 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:29.670782    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08e9108a-8670-47b9-a15b-a530895231e1" path="/var/lib/kubelet/pods/08e9108a-8670-47b9-a15b-a530895231e1/volumes"
Nov 03 21:39:33.064961 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:33.064920    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a3ef192-79tzf"]
Nov 03 21:39:33.111903 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:33.111590    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a3ef192-79tzf"]
Nov 03 21:39:33.669248 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:33.669197    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="243d8a52-c0be-4581-ab64-bf3897f88f1d" path="/var/lib/kubelet/pods/243d8a52-c0be-4581-ab64-bf3897f88f1d/volumes"
Nov 03 21:39:40.046740 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:40.046704    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23bb161c-sc4wg"]
Nov 03 21:39:40.095914 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:40.095876    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23bb161c-sc4wg"]
Nov 03 21:39:40.930094 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:39:40.930055    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:39:41.668248 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:41.668211    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="698a2a54-5236-49a9-b759-7659d7712b1c" path="/var/lib/kubelet/pods/698a2a54-5236-49a9-b759-7659d7712b1c/volumes"
Nov 03 21:39:45.896607 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:45.896576    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 21:39:45.905072 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:45.905041    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:39:45.912122 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:45.912098    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 21:39:45.920928 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:45.920900    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:39:46.320164 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:46.320125    5053 scope.go:122] "RemoveContainer" containerID="218f3678c71e2116d6cd5e320ae1ebfe698f1dbdadc9e0483e62d462fbea5300"
Nov 03 21:39:46.332892 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:46.332868    5053 scope.go:122] "RemoveContainer" containerID="89637fd5592a5e0eaa04e9fbb9bb530577b22a6388c824186662b4f5ac8c77eb"
Nov 03 21:39:46.348336 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:46.348305    5053 scope.go:122] "RemoveContainer" containerID="f0c40b2358feab6b720c07e6ee513cb399677a1a39898041adf158f12d0abc5a"
Nov 03 21:39:46.359544 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:46.359520    5053 scope.go:122] "RemoveContainer" containerID="60f30c6f8f4b9b0d1e7ac782e7f2dc6fd263b226478f3814269fb3e01dd34a30"
Nov 03 21:39:46.369391 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:46.369370    5053 scope.go:122] "RemoveContainer" containerID="56fb26aad760ca958f111bb5fe00a34b60dd0f022c1dd161a32705578c599e75"
Nov 03 21:39:52.051595 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:52.051553    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4a70c76-lqrt8"]
Nov 03 21:39:52.101002 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:52.100965    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4a70c76-lqrt8"]
Nov 03 21:39:53.669442 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:53.669408    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62eb6f28-db58-479d-9e7f-11d756f8fe0d" path="/var/lib/kubelet/pods/62eb6f28-db58-479d-9e7f-11d756f8fe0d/volumes"
Nov 03 21:39:55.316045 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.316006    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"]
Nov 03 21:39:55.317084 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.317063    5053 cpu_manager.go:414] "containerMap: removing container" podUID="5553e95b-04c9-4716-a5c6-8f5cdf96bbd2" containerName="payload-pre-processing"
Nov 03 21:39:55.317168 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.317085    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5553e95b-04c9-4716-a5c6-8f5cdf96bbd2" containerName="payload-pre-processing"
Nov 03 21:39:55.317168 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.317103    5053 cpu_manager.go:414] "containerMap: removing container" podUID="0ead33cd-8e19-422f-bef3-4e069ba97eda" containerName="payload-processing"
Nov 03 21:39:55.317168 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.317111    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ead33cd-8e19-422f-bef3-4e069ba97eda" containerName="payload-processing"
Nov 03 21:39:55.317382 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.317170    5053 cpu_manager.go:414] "containerMap: removing container" podUID="1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f" containerName="revoke-keys"
Nov 03 21:39:55.317382 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.317177    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f" containerName="revoke-keys"
Nov 03 21:39:55.317487 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.317387    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f" containerName="revoke-keys"
Nov 03 21:39:55.317487 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.317406    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="5553e95b-04c9-4716-a5c6-8f5cdf96bbd2" containerName="payload-pre-processing"
Nov 03 21:39:55.317487 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.317414    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ead33cd-8e19-422f-bef3-4e069ba97eda" containerName="payload-processing"
Nov 03 21:39:55.321415 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.321393    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.391261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.389820    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-85d7ff-openshift-default-dockercfg-lzv4z\""
Nov 03 21:39:55.406259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.395226    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-envoy\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.406259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.395325    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istiod-ca-cert\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.406259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.395423    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-workload-socket\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.406259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.395521    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-credential-socket\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.406259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.395581    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrzkf\" (UniqueName: \"kubernetes.io/projected/5a7feda9-6f61-4bf7-8561-05fec79b8b17-kube-api-access-hrzkf\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.406259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.395643    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-token\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.406259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.395745    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-workload-certs\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.406259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.395837    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-podinfo\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.406259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.395905    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-data\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.497265 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.495915    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"]
Nov 03 21:39:55.505282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.498332    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istiod-ca-cert\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.505282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.498437    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-workload-socket\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.505282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.498537    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-credential-socket\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.505282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.498577    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrzkf\" (UniqueName: \"kubernetes.io/projected/5a7feda9-6f61-4bf7-8561-05fec79b8b17-kube-api-access-hrzkf\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.505282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.498630    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-token\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.505282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.498712    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-workload-certs\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.505282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.498793    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-podinfo\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.505282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.498862    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-data\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.505282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.498917    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-envoy\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.505282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.500638    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istiod-ca-cert\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.505282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.500909    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-workload-socket\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.505282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.501148    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-credential-socket\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.505282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.503087    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-workload-certs\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.505282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.503347    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-data\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.542016 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.541982    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-podinfo\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.542791 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.542766    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-envoy\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.563295 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.563093    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-token\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.563901 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.563874    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrzkf\" (UniqueName: \"kubernetes.io/projected/5a7feda9-6f61-4bf7-8561-05fec79b8b17-kube-api-access-hrzkf\") pod \"e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") " pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:55.638898 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:55.638769    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:56.091462 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:39:56.091432    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a7feda9_6f61_4bf7_8561_05fec79b8b17.slice/crio-309469604e27229d8735536b81b57d81faea8e43056b095f3256497fec8ad1c2 WatchSource:0}: Error finding container 309469604e27229d8735536b81b57d81faea8e43056b095f3256497fec8ad1c2: Status 404 returned error can't find the container with id 309469604e27229d8735536b81b57d81faea8e43056b095f3256497fec8ad1c2
Nov 03 21:39:56.091735 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:56.091715    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"]
Nov 03 21:39:56.094151 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:56.094117    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:39:56.094228 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:56.094190    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:39:56.094228 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:56.094220    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:39:56.350065 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:56.349960    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg" event={"ID":"5a7feda9-6f61-4bf7-8561-05fec79b8b17","Type":"ContainerStarted","Data":"d817e5c652e1a0db2a2ea8d374f2d359ffb45317185a4a80e10c3efe9044d196"}
Nov 03 21:39:56.350484 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:56.350060    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg" event={"ID":"5a7feda9-6f61-4bf7-8561-05fec79b8b17","Type":"ContainerStarted","Data":"309469604e27229d8735536b81b57d81faea8e43056b095f3256497fec8ad1c2"}
Nov 03 21:39:56.472290 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:56.469290    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg" podStartSLOduration=1.469230532 podStartE2EDuration="1.469230532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:56.468542436 +0000 UTC m=+1510.962464350" watchObservedRunningTime="2026-11-03 21:39:56.469230532 +0000 UTC m=+1510.963152409"
Nov 03 21:39:56.639178 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:56.639094    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:57.048207 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:57.048163    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9e7ee1-dbdgz"]
Nov 03 21:39:57.052425 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:57.052394    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9e7ee1-dbdgz"]
Nov 03 21:39:57.645535 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:57.645504    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:57.668688 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:57.668657    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89850b47-7092-4eed-a237-72cc22796def" path="/var/lib/kubelet/pods/89850b47-7092-4eed-a237-72cc22796def/volumes"
Nov 03 21:39:58.368019 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:58.367988    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:39:58.369032 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:39:58.369007    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:40:00.047859 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:00.047826    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b33a1c-lmj2b"]
Nov 03 21:40:00.096711 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:00.096678    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b33a1c-lmj2b"]
Nov 03 21:40:01.668399 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:01.668365    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4f15d8b-0518-4a43-b22f-508457222b3b" path="/var/lib/kubelet/pods/e4f15d8b-0518-4a43-b22f-508457222b3b/volumes"
Nov 03 21:40:07.050140 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:07.050106    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f8f073b-94nrl"]
Nov 03 21:40:07.097631 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:07.097597    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f8f073b-94nrl"]
Nov 03 21:40:07.668974 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:07.668944    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="253e3305-deb7-4478-844b-387347d69de0" path="/var/lib/kubelet/pods/253e3305-deb7-4478-844b-387347d69de0/volumes"
Nov 03 21:40:10.929456 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:40:10.929418    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:40:18.280145 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:18.280110    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-29s5p"]
Nov 03 21:40:18.331696 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:18.331639    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-29s5p"]
Nov 03 21:40:19.668015 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:19.667983    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba223252-af37-4d5f-b5c0-ccb1e1f91c05" path="/var/lib/kubelet/pods/ba223252-af37-4d5f-b5c0-ccb1e1f91c05/volumes"
Nov 03 21:40:21.046755 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:21.046722    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4abd935f-ffjcj"]
Nov 03 21:40:21.094027 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:21.093996    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4abd935f-ffjcj"]
Nov 03 21:40:21.668379 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:21.668345    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58de2b96-1689-48bb-a906-cf768631fb00" path="/var/lib/kubelet/pods/58de2b96-1689-48bb-a906-cf768631fb00/volumes"
Nov 03 21:40:22.048519 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:22.048485    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-182aff01-lfffm"]
Nov 03 21:40:22.097961 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:22.097930    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-182aff01-lfffm"]
Nov 03 21:40:23.668701 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:23.668667    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="594ecf22-3e9b-4566-a79c-be4dd2b1d85c" path="/var/lib/kubelet/pods/594ecf22-3e9b-4566-a79c-be4dd2b1d85c/volumes"
Nov 03 21:40:40.047142 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:40.047103    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c325017c-qkr55"]
Nov 03 21:40:40.096211 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:40.096172    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c325017c-qkr55"]
Nov 03 21:40:40.429019 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:40.428920    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29472d-qzbth"]
Nov 03 21:40:40.433904 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:40.433881    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29472d-qzbth"
Nov 03 21:40:40.473687 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:40.473652    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-29472d-qzbth\" (UID: \"6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29472d-qzbth"
Nov 03 21:40:40.473942 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:40.473921    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-29472d-qzbth\" (UID: \"6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29472d-qzbth"
Nov 03 21:40:40.474033 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:40.474019    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln6vj\" (UniqueName: \"kubernetes.io/projected/6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0-kube-api-access-ln6vj\") pod \"maas-api-revoke-keys-e2e-api-a-29472d-qzbth\" (UID: \"6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29472d-qzbth"
Nov 03 21:40:40.490526 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:40.490493    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29472d-qzbth"]
Nov 03 21:40:40.533693 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:40.533654    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:40:40.533693 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:40.533669    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:40:40.533693 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:40.533671    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:40:40.535628 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:40.535602    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:40:40.575537 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:40.575506    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-29472d-qzbth\" (UID: \"6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29472d-qzbth"
Nov 03 21:40:40.575791 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:40.575591    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln6vj\" (UniqueName: \"kubernetes.io/projected/6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0-kube-api-access-ln6vj\") pod \"maas-api-revoke-keys-e2e-api-a-29472d-qzbth\" (UID: \"6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29472d-qzbth"
Nov 03 21:40:40.575791 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:40.575645    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-29472d-qzbth\" (UID: \"6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29472d-qzbth"
Nov 03 21:40:40.576459 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:40.576433    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-29472d-qzbth\" (UID: \"6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29472d-qzbth"
Nov 03 21:40:40.579344 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:40.579320    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-29472d-qzbth\" (UID: \"6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29472d-qzbth"
Nov 03 21:40:40.632492 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:40.632458    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln6vj\" (UniqueName: \"kubernetes.io/projected/6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0-kube-api-access-ln6vj\") pod \"maas-api-revoke-keys-e2e-api-a-29472d-qzbth\" (UID: \"6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29472d-qzbth"
Nov 03 21:40:40.749555 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:40.749523    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29472d-qzbth"
Nov 03 21:40:40.960142 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:40:40.960105    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:40:41.154882 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:40:41.154854    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e8affbd_e66d_4ba0_a1e6_bd974f28d3b0.slice/crio-e983a7f3860ae0f8bfc68171eac409dee612e1858472f34233c2f792e5fee1c8 WatchSource:0}: Error finding container e983a7f3860ae0f8bfc68171eac409dee612e1858472f34233c2f792e5fee1c8: Status 404 returned error can't find the container with id e983a7f3860ae0f8bfc68171eac409dee612e1858472f34233c2f792e5fee1c8
Nov 03 21:40:41.155360 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:41.155336    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29472d-qzbth"]
Nov 03 21:40:41.680758 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:41.680712    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d606955a-bcbf-44cc-9d76-7601ad0b3e94" path="/var/lib/kubelet/pods/d606955a-bcbf-44cc-9d76-7601ad0b3e94/volumes"
Nov 03 21:40:41.941932 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:41.941823    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29472d-qzbth" event={"ID":"6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0","Type":"ContainerStarted","Data":"0d1bf81374affda12d88632382413e57c6cce38eb93e88ba1c6f562867acdeff"}
Nov 03 21:40:41.941932 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:41.941890    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29472d-qzbth" event={"ID":"6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0","Type":"ContainerStarted","Data":"e983a7f3860ae0f8bfc68171eac409dee612e1858472f34233c2f792e5fee1c8"}
Nov 03 21:40:42.054982 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:42.054912    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29472d-qzbth" podStartSLOduration=2.054893773 podStartE2EDuration="2.054893773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:40:42.053691554 +0000 UTC m=+1556.547613441" watchObservedRunningTime="2026-11-03 21:40:42.054893773 +0000 UTC m=+1556.548815649"
Nov 03 21:40:42.952101 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:42.952070    5053 generic.go:356] "Generic (PLEG): container finished" podID="6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0" containerID="0d1bf81374affda12d88632382413e57c6cce38eb93e88ba1c6f562867acdeff" exitCode=0
Nov 03 21:40:42.952549 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:42.952148    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29472d-qzbth" event={"ID":"6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0","Type":"ContainerDied","Data":"0d1bf81374affda12d88632382413e57c6cce38eb93e88ba1c6f562867acdeff"}
Nov 03 21:40:44.166571 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:44.166546    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29472d-qzbth"
Nov 03 21:40:44.249171 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:44.249088    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0-kube-api-access-ln6vj\" (UniqueName: \"kubernetes.io/projected/6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0-kube-api-access-ln6vj\") pod \"6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0\" (UID: \"6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0\") "
Nov 03 21:40:44.249373 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:44.249281    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0-maas-api-openshift-ca\") pod \"6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0\" (UID: \"6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0\") "
Nov 03 21:40:44.249454 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:44.249399    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0-maas-api-xks-ca\") pod \"6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0\" (UID: \"6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0\") "
Nov 03 21:40:44.249672 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:44.249646    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0-maas-api-openshift-ca" pod "6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0" (UID: "6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:40:44.249931 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:44.249913    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:40:44.251518 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:44.251488    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0-kube-api-access-ln6vj" pod "6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0" (UID: "6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0"). InnerVolumeSpecName "kube-api-access-ln6vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:40:44.251820 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:44.251800    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0-maas-api-xks-ca" pod "6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0" (UID: "6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:40:44.351513 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:44.351481    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:40:44.351513 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:44.351511    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln6vj\" (UniqueName: \"kubernetes.io/projected/6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0-kube-api-access-ln6vj\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:40:44.974958 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:44.974917    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29472d-qzbth"
Nov 03 21:40:44.974958 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:44.974951    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29472d-qzbth" event={"ID":"6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0","Type":"ContainerDied","Data":"e983a7f3860ae0f8bfc68171eac409dee612e1858472f34233c2f792e5fee1c8"}
Nov 03 21:40:44.975178 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:44.975003    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e983a7f3860ae0f8bfc68171eac409dee612e1858472f34233c2f792e5fee1c8"
Nov 03 21:40:46.438478 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:46.438438    5053 scope.go:122] "RemoveContainer" containerID="1230465ecd422e6297a2ce3104b0ada508aa8155c8c658fb5c5fabe2b4a13c9a"
Nov 03 21:40:46.450126 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:46.449976    5053 scope.go:122] "RemoveContainer" containerID="d3eb856724542f530cd59ee0abc012586480ee105043156ab778a42e292ccca6"
Nov 03 21:40:46.463280 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:46.463164    5053 scope.go:122] "RemoveContainer" containerID="39c78fda274361b10fee3e809efcfad9c7112c440b552d30a2876f80c8a2fb9c"
Nov 03 21:40:46.476556 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:46.476497    5053 scope.go:122] "RemoveContainer" containerID="1bc1c501c9ea9b6c6b5215adc2c7d760b2237126b7edb56d33332b60989474bb"
Nov 03 21:40:46.488507 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:46.488483    5053 scope.go:122] "RemoveContainer" containerID="8dff7203d1ffca7c345584865a05c3bcb613fa050a78b315852338f160a8b22d"
Nov 03 21:40:46.501946 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:46.501924    5053 scope.go:122] "RemoveContainer" containerID="ca94d8d178f302425e637be85e28e140ec1366bd4fbd22358304a45e2c9b599c"
Nov 03 21:40:46.511784 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:46.511764    5053 scope.go:122] "RemoveContainer" containerID="5fefee905ea404b84bc89925f77cdda5eeecd5bea188bb94592d1a85956c407f"
Nov 03 21:40:47.371781 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:47.371015    5053 patch_prober.go:29] interesting pod/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": read tcp 10.132.0.2:42962->10.132.0.146:15021: read: connection reset by peer" start-of-body=
Nov 03 21:40:47.371781 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:47.371110    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq" podUID="83a53cc7-96e4-49db-a8ac-2d2de9e84188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": read tcp 10.132.0.2:42962->10.132.0.146:15021: read: connection reset by peer"
Nov 03 21:40:48.062260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:48.061896    5053 generic.go:356] "Generic (PLEG): container finished" podID="83a53cc7-96e4-49db-a8ac-2d2de9e84188" containerID="16be4c649a9cbd5baa803ba185f0dcc51e312bf4e55767a02b20d64a20d72882" exitCode=0
Nov 03 21:40:48.062260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:48.061974    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq" event={"ID":"83a53cc7-96e4-49db-a8ac-2d2de9e84188","Type":"ContainerDied","Data":"16be4c649a9cbd5baa803ba185f0dcc51e312bf4e55767a02b20d64a20d72882"}
Nov 03 21:40:48.062833 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:48.062484    5053 scope.go:122] "RemoveContainer" containerID="16be4c649a9cbd5baa803ba185f0dcc51e312bf4e55767a02b20d64a20d72882"
Nov 03 21:40:48.071858 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:48.065733    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:40:48.071858 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:48.065824    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:40:48.071858 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:48.065867    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:40:49.073036 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:49.072998    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq" event={"ID":"83a53cc7-96e4-49db-a8ac-2d2de9e84188","Type":"ContainerStarted","Data":"20c18c028d05a29a3d45f681554e95cf34d5bbe9071132ec3225acad81c13c2c"}
Nov 03 21:40:49.442543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:49.442458    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:40:49.442773 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:49.442750    5053 patch_prober.go:29] interesting pod/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 03 21:40:49.442946 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:49.442918    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq" podUID="83a53cc7-96e4-49db-a8ac-2d2de9e84188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 03 21:40:50.439665 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:50.439627    5053 patch_prober.go:29] interesting pod/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 03 21:40:50.440315 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:50.440282    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq" podUID="83a53cc7-96e4-49db-a8ac-2d2de9e84188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 03 21:40:51.439201 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:51.439163    5053 patch_prober.go:29] interesting pod/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 03 21:40:51.439381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:51.439219    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq" podUID="83a53cc7-96e4-49db-a8ac-2d2de9e84188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 03 21:40:52.128263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:52.128030    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f"]
Nov 03 21:40:52.134348 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:52.134309    5053 cpu_manager.go:414] "containerMap: removing container" podUID="6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0" containerName="revoke-keys"
Nov 03 21:40:52.134348 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:52.134339    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0" containerName="revoke-keys"
Nov 03 21:40:52.134594 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:52.134581    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0" containerName="revoke-keys"
Nov 03 21:40:52.137903 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:52.137881    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f"
Nov 03 21:40:52.169932 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:52.169900    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f"]
Nov 03 21:40:52.222262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:52.221548    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:40:52.222262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:52.221648    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:40:52.222262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:52.221700    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:40:52.222262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:52.221801    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:40:52.247631 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:52.247584    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d8a1267-4c43-4f54-815d-79e78fef26a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f\" (UID: \"2d8a1267-4c43-4f54-815d-79e78fef26a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f"
Nov 03 21:40:52.247948 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:52.247922    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d8a1267-4c43-4f54-815d-79e78fef26a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f\" (UID: \"2d8a1267-4c43-4f54-815d-79e78fef26a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f"
Nov 03 21:40:52.248220 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:52.248200    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqdlp\" (UniqueName: \"kubernetes.io/projected/2d8a1267-4c43-4f54-815d-79e78fef26a5-kube-api-access-mqdlp\") pod \"maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f\" (UID: \"2d8a1267-4c43-4f54-815d-79e78fef26a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f"
Nov 03 21:40:52.349503 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:52.349467    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d8a1267-4c43-4f54-815d-79e78fef26a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f\" (UID: \"2d8a1267-4c43-4f54-815d-79e78fef26a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f"
Nov 03 21:40:52.349715 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:52.349619    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqdlp\" (UniqueName: \"kubernetes.io/projected/2d8a1267-4c43-4f54-815d-79e78fef26a5-kube-api-access-mqdlp\") pod \"maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f\" (UID: \"2d8a1267-4c43-4f54-815d-79e78fef26a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f"
Nov 03 21:40:52.349853 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:52.349837    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d8a1267-4c43-4f54-815d-79e78fef26a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f\" (UID: \"2d8a1267-4c43-4f54-815d-79e78fef26a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f"
Nov 03 21:40:52.350447 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:52.350402    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d8a1267-4c43-4f54-815d-79e78fef26a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f\" (UID: \"2d8a1267-4c43-4f54-815d-79e78fef26a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f"
Nov 03 21:40:52.352139 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:52.352116    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d8a1267-4c43-4f54-815d-79e78fef26a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f\" (UID: \"2d8a1267-4c43-4f54-815d-79e78fef26a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f"
Nov 03 21:40:52.427034 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:52.426918    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqdlp\" (UniqueName: \"kubernetes.io/projected/2d8a1267-4c43-4f54-815d-79e78fef26a5-kube-api-access-mqdlp\") pod \"maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f\" (UID: \"2d8a1267-4c43-4f54-815d-79e78fef26a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f"
Nov 03 21:40:52.439119 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:52.439086    5053 patch_prober.go:29] interesting pod/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 03 21:40:52.439335 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:52.439170    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq" podUID="83a53cc7-96e4-49db-a8ac-2d2de9e84188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 03 21:40:52.451965 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:52.451926    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f"
Nov 03 21:40:52.878948 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:52.878917    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f"]
Nov 03 21:40:53.140431 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:53.140340    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f" event={"ID":"2d8a1267-4c43-4f54-815d-79e78fef26a5","Type":"ContainerStarted","Data":"ad4a0a285c1131541694aaa23e68e0c743ab3216db0d2822efe54182871a8713"}
Nov 03 21:40:53.439302 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:53.439273    5053 patch_prober.go:29] interesting pod/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 03 21:40:53.439493 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:53.439463    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq" podUID="83a53cc7-96e4-49db-a8ac-2d2de9e84188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 03 21:40:54.178364 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:54.178332    5053 generic.go:356] "Generic (PLEG): container finished" podID="2d8a1267-4c43-4f54-815d-79e78fef26a5" containerID="1ebdeabedab65cccbaaa91cf46fc8090368e1634325addb6c8196d0a733da5d6" exitCode=0
Nov 03 21:40:54.178857 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:54.178582    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f" event={"ID":"2d8a1267-4c43-4f54-815d-79e78fef26a5","Type":"ContainerDied","Data":"1ebdeabedab65cccbaaa91cf46fc8090368e1634325addb6c8196d0a733da5d6"}
Nov 03 21:40:54.348172 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:54.348078    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr"]
Nov 03 21:40:54.436619 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:54.436584    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs"]
Nov 03 21:40:54.439691 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:54.439652    5053 patch_prober.go:29] interesting pod/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 03 21:40:54.439879 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:54.439702    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq" podUID="83a53cc7-96e4-49db-a8ac-2d2de9e84188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 03 21:40:55.368094 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:55.368064    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f"
Nov 03 21:40:55.443107 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:55.443082    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:40:55.443522 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:55.443500    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:40:55.444286 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:55.444228    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:40:55.508648 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:55.508612    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d8a1267-4c43-4f54-815d-79e78fef26a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d8a1267-4c43-4f54-815d-79e78fef26a5-maas-api-openshift-ca\") pod \"2d8a1267-4c43-4f54-815d-79e78fef26a5\" (UID: \"2d8a1267-4c43-4f54-815d-79e78fef26a5\") "
Nov 03 21:40:55.508863 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:55.508730    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8a1267-4c43-4f54-815d-79e78fef26a5-kube-api-access-mqdlp\" (UniqueName: \"kubernetes.io/projected/2d8a1267-4c43-4f54-815d-79e78fef26a5-kube-api-access-mqdlp\") pod \"2d8a1267-4c43-4f54-815d-79e78fef26a5\" (UID: \"2d8a1267-4c43-4f54-815d-79e78fef26a5\") "
Nov 03 21:40:55.508937 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:55.508903    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d8a1267-4c43-4f54-815d-79e78fef26a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d8a1267-4c43-4f54-815d-79e78fef26a5-maas-api-xks-ca\") pod \"2d8a1267-4c43-4f54-815d-79e78fef26a5\" (UID: \"2d8a1267-4c43-4f54-815d-79e78fef26a5\") "
Nov 03 21:40:55.509053 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:55.509023    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d8a1267-4c43-4f54-815d-79e78fef26a5-maas-api-openshift-ca" pod "2d8a1267-4c43-4f54-815d-79e78fef26a5" (UID: "2d8a1267-4c43-4f54-815d-79e78fef26a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:40:55.509843 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:55.509823    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d8a1267-4c43-4f54-815d-79e78fef26a5-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:40:55.511365 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:55.511342    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8a1267-4c43-4f54-815d-79e78fef26a5-kube-api-access-mqdlp" pod "2d8a1267-4c43-4f54-815d-79e78fef26a5" (UID: "2d8a1267-4c43-4f54-815d-79e78fef26a5"). InnerVolumeSpecName "kube-api-access-mqdlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:40:55.511840 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:55.511812    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d8a1267-4c43-4f54-815d-79e78fef26a5-maas-api-xks-ca" pod "2d8a1267-4c43-4f54-815d-79e78fef26a5" (UID: "2d8a1267-4c43-4f54-815d-79e78fef26a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:40:55.611564 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:55.611524    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d8a1267-4c43-4f54-815d-79e78fef26a5-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:40:55.611564 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:55.611559    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqdlp\" (UniqueName: \"kubernetes.io/projected/2d8a1267-4c43-4f54-815d-79e78fef26a5-kube-api-access-mqdlp\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:40:56.195685 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:56.195657    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f"
Nov 03 21:40:56.195891 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:56.195691    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f" event={"ID":"2d8a1267-4c43-4f54-815d-79e78fef26a5","Type":"ContainerDied","Data":"ad4a0a285c1131541694aaa23e68e0c743ab3216db0d2822efe54182871a8713"}
Nov 03 21:40:56.195891 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:56.195738    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad4a0a285c1131541694aaa23e68e0c743ab3216db0d2822efe54182871a8713"
Nov 03 21:40:59.348813 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:59.348772    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr" podUID="e13ebd3d-b7a9-43d9-a490-4e6d37b573d7" containerName="payload-processing" containerID="cri-o://16dfd38fc8d926a39f12386f57f0384e7cb2d2ed8b2d2357b4e63a089e998703" gracePeriod=25
Nov 03 21:40:59.437578 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:59.437532    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs" podUID="7650dde2-539f-4535-b69f-8e90820be009" containerName="payload-pre-processing" containerID="cri-o://eefb5077bd928545ce3514f34f6f050316ba9ed2a6596520c0e264472e342555" gracePeriod=25
Nov 03 21:40:59.759484 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:59.759456    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr"
Nov 03 21:40:59.802211 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:59.802181    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs"
Nov 03 21:40:59.888881 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:59.888797    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13ebd3d-b7a9-43d9-a490-4e6d37b573d7-kube-api-access-6w9sz\" (UniqueName: \"kubernetes.io/projected/e13ebd3d-b7a9-43d9-a490-4e6d37b573d7-kube-api-access-6w9sz\") pod \"e13ebd3d-b7a9-43d9-a490-4e6d37b573d7\" (UID: \"e13ebd3d-b7a9-43d9-a490-4e6d37b573d7\") "
Nov 03 21:40:59.889052 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:59.888989    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7650dde2-539f-4535-b69f-8e90820be009-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7650dde2-539f-4535-b69f-8e90820be009-plugins-config-volume\") pod \"7650dde2-539f-4535-b69f-8e90820be009\" (UID: \"7650dde2-539f-4535-b69f-8e90820be009\") "
Nov 03 21:40:59.889131 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:59.889119    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7650dde2-539f-4535-b69f-8e90820be009-kube-api-access-mcddc\" (UniqueName: \"kubernetes.io/projected/7650dde2-539f-4535-b69f-8e90820be009-kube-api-access-mcddc\") pod \"7650dde2-539f-4535-b69f-8e90820be009\" (UID: \"7650dde2-539f-4535-b69f-8e90820be009\") "
Nov 03 21:40:59.889286 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:59.889267    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e13ebd3d-b7a9-43d9-a490-4e6d37b573d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13ebd3d-b7a9-43d9-a490-4e6d37b573d7-plugins-config-volume\") pod \"e13ebd3d-b7a9-43d9-a490-4e6d37b573d7\" (UID: \"e13ebd3d-b7a9-43d9-a490-4e6d37b573d7\") "
Nov 03 21:40:59.889449 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:59.889398    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7650dde2-539f-4535-b69f-8e90820be009-plugins-config-volume" pod "7650dde2-539f-4535-b69f-8e90820be009" (UID: "7650dde2-539f-4535-b69f-8e90820be009"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:40:59.889630 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:59.889607    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13ebd3d-b7a9-43d9-a490-4e6d37b573d7-plugins-config-volume" pod "e13ebd3d-b7a9-43d9-a490-4e6d37b573d7" (UID: "e13ebd3d-b7a9-43d9-a490-4e6d37b573d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:40:59.890050 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:59.890031    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13ebd3d-b7a9-43d9-a490-4e6d37b573d7-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:40:59.890110 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:59.890058    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7650dde2-539f-4535-b69f-8e90820be009-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:40:59.891967 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:59.891939    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13ebd3d-b7a9-43d9-a490-4e6d37b573d7-kube-api-access-6w9sz" pod "e13ebd3d-b7a9-43d9-a490-4e6d37b573d7" (UID: "e13ebd3d-b7a9-43d9-a490-4e6d37b573d7"). InnerVolumeSpecName "kube-api-access-6w9sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:40:59.892056 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:59.891987    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7650dde2-539f-4535-b69f-8e90820be009-kube-api-access-mcddc" pod "7650dde2-539f-4535-b69f-8e90820be009" (UID: "7650dde2-539f-4535-b69f-8e90820be009"). InnerVolumeSpecName "kube-api-access-mcddc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:40:59.991608 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:59.991571    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w9sz\" (UniqueName: \"kubernetes.io/projected/e13ebd3d-b7a9-43d9-a490-4e6d37b573d7-kube-api-access-6w9sz\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:40:59.991608 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:40:59.991610    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcddc\" (UniqueName: \"kubernetes.io/projected/7650dde2-539f-4535-b69f-8e90820be009-kube-api-access-mcddc\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:00.233639 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:00.233600    5053 generic.go:356] "Generic (PLEG): container finished" podID="e13ebd3d-b7a9-43d9-a490-4e6d37b573d7" containerID="16dfd38fc8d926a39f12386f57f0384e7cb2d2ed8b2d2357b4e63a089e998703" exitCode=0
Nov 03 21:41:00.233639 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:00.233641    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr"
Nov 03 21:41:00.233866 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:00.233686    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr" event={"ID":"e13ebd3d-b7a9-43d9-a490-4e6d37b573d7","Type":"ContainerDied","Data":"16dfd38fc8d926a39f12386f57f0384e7cb2d2ed8b2d2357b4e63a089e998703"}
Nov 03 21:41:00.233866 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:00.233741    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr" event={"ID":"e13ebd3d-b7a9-43d9-a490-4e6d37b573d7","Type":"ContainerDied","Data":"8c2d5c40c7f1355cefc14e48bb87de6fd2377bd5a5425fb54f97a053857c82a8"}
Nov 03 21:41:00.233866 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:00.233776    5053 scope.go:122] "RemoveContainer" containerID="16dfd38fc8d926a39f12386f57f0384e7cb2d2ed8b2d2357b4e63a089e998703"
Nov 03 21:41:00.235281 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:00.235256    5053 generic.go:356] "Generic (PLEG): container finished" podID="7650dde2-539f-4535-b69f-8e90820be009" containerID="eefb5077bd928545ce3514f34f6f050316ba9ed2a6596520c0e264472e342555" exitCode=0
Nov 03 21:41:00.235410 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:00.235313    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs"
Nov 03 21:41:00.235410 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:00.235349    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs" event={"ID":"7650dde2-539f-4535-b69f-8e90820be009","Type":"ContainerDied","Data":"eefb5077bd928545ce3514f34f6f050316ba9ed2a6596520c0e264472e342555"}
Nov 03 21:41:00.235410 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:00.235395    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs" event={"ID":"7650dde2-539f-4535-b69f-8e90820be009","Type":"ContainerDied","Data":"8fbcc6a001b27277158431c42e1af668a6425f02b4516e0a43ac3270f584a215"}
Nov 03 21:41:00.247916 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:00.247871    5053 scope.go:122] "RemoveContainer" containerID="16dfd38fc8d926a39f12386f57f0384e7cb2d2ed8b2d2357b4e63a089e998703"
Nov 03 21:41:00.248217 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:41:00.248198    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16dfd38fc8d926a39f12386f57f0384e7cb2d2ed8b2d2357b4e63a089e998703\": container with ID starting with 16dfd38fc8d926a39f12386f57f0384e7cb2d2ed8b2d2357b4e63a089e998703 not found: ID does not exist" containerID="16dfd38fc8d926a39f12386f57f0384e7cb2d2ed8b2d2357b4e63a089e998703"
Nov 03 21:41:00.248365 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:00.248227    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16dfd38fc8d926a39f12386f57f0384e7cb2d2ed8b2d2357b4e63a089e998703"} err="failed to get container status \"16dfd38fc8d926a39f12386f57f0384e7cb2d2ed8b2d2357b4e63a089e998703\": rpc error: code = NotFound desc = could not find container \"16dfd38fc8d926a39f12386f57f0384e7cb2d2ed8b2d2357b4e63a089e998703\": container with ID starting with 16dfd38fc8d926a39f12386f57f0384e7cb2d2ed8b2d2357b4e63a089e998703 not found: ID does not exist"
Nov 03 21:41:00.248365 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:00.248267    5053 scope.go:122] "RemoveContainer" containerID="eefb5077bd928545ce3514f34f6f050316ba9ed2a6596520c0e264472e342555"
Nov 03 21:41:00.259351 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:00.259330    5053 scope.go:122] "RemoveContainer" containerID="eefb5077bd928545ce3514f34f6f050316ba9ed2a6596520c0e264472e342555"
Nov 03 21:41:00.259681 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:41:00.259661    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eefb5077bd928545ce3514f34f6f050316ba9ed2a6596520c0e264472e342555\": container with ID starting with eefb5077bd928545ce3514f34f6f050316ba9ed2a6596520c0e264472e342555 not found: ID does not exist" containerID="eefb5077bd928545ce3514f34f6f050316ba9ed2a6596520c0e264472e342555"
Nov 03 21:41:00.259768 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:00.259691    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eefb5077bd928545ce3514f34f6f050316ba9ed2a6596520c0e264472e342555"} err="failed to get container status \"eefb5077bd928545ce3514f34f6f050316ba9ed2a6596520c0e264472e342555\": rpc error: code = NotFound desc = could not find container \"eefb5077bd928545ce3514f34f6f050316ba9ed2a6596520c0e264472e342555\": container with ID starting with eefb5077bd928545ce3514f34f6f050316ba9ed2a6596520c0e264472e342555 not found: ID does not exist"
Nov 03 21:41:00.397836 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:00.397802    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr"]
Nov 03 21:41:00.402007 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:00.401977    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-29472d-fdf48f66b-vt2lr"]
Nov 03 21:41:00.510722 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:00.510633    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs"]
Nov 03 21:41:00.516088 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:00.516050    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-29472d-7ff8fcf4c9-g4wzs"]
Nov 03 21:41:01.668936 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:01.668905    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7650dde2-539f-4535-b69f-8e90820be009" path="/var/lib/kubelet/pods/7650dde2-539f-4535-b69f-8e90820be009/volumes"
Nov 03 21:41:01.669463 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:01.669448    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e13ebd3d-b7a9-43d9-a490-4e6d37b573d7" path="/var/lib/kubelet/pods/e13ebd3d-b7a9-43d9-a490-4e6d37b573d7/volumes"
Nov 03 21:41:03.881562 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:03.881529    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2"]
Nov 03 21:41:03.988175 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:03.988141    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9"]
Nov 03 21:41:06.311005 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:06.310967    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"]
Nov 03 21:41:06.311538 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:06.311278    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd" podUID="51d17433-dc86-491b-9276-080c84dfdc8e" containerName="istio-proxy" containerID="cri-o://b2d879b06a72cda2dd3d60517e935ff43844f9713ce2f1249ab29dcb6331a45b" gracePeriod=30
Nov 03 21:41:06.360480 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:06.360443    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"]
Nov 03 21:41:06.360680 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:06.360643    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc" podUID="c43adf43-1729-4f0d-b6c7-3918c0fff604" containerName="istio-proxy" containerID="cri-o://3b74edf2f27c8fddc02faf0dc29fa92b58b753482abcc5f5fc886630bd90214e" gracePeriod=30
Nov 03 21:41:06.378559 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:06.378519    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"]
Nov 03 21:41:06.378833 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:06.378807    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z" podUID="b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa" containerName="istio-proxy" containerID="cri-o://c9ae10655d87ddcaea7236fc2fb980e11c5677bfb8a3addcc57f4b6edafe51b8" gracePeriod=30
Nov 03 21:41:06.413322 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:06.413267    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"]
Nov 03 21:41:06.413557 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:06.413527    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k" podUID="b9a77c28-78e6-42ce-89cd-e9956e13bfc2" containerName="istio-proxy" containerID="cri-o://565a74b41c0a9321ac8cdc1edbbadcdde952cfc70d0132c73a76748f5ac113fc" gracePeriod=30
Nov 03 21:41:06.867577 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:06.867542    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:41:06.867920 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:06.867871    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd" podUID="51d17433-dc86-491b-9276-080c84dfdc8e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:41:07.155418 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:07.155331    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"]
Nov 03 21:41:07.155571 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:07.155552    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh" podUID="18feb2eb-b1a3-492c-8699-e507bf82b55f" containerName="istio-proxy" containerID="cri-o://40aac3ccec408709b293c389e67206c2382e7b8ab290222277764b399f8b8669" gracePeriod=30
Nov 03 21:41:07.160358 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:07.160332    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"]
Nov 03 21:41:07.160551 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:07.160531    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6" podUID="7b1d9743-441e-4e3a-a42b-c521bd99b7c3" containerName="istio-proxy" containerID="cri-o://c347183848d9e24f307629b3a84b05c9e6a9140a6a2bbfc77b5df89587e71004" gracePeriod=30
Nov 03 21:41:08.883290 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:08.883212    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2" podUID="57c5d5d3-f260-46e3-9ea8-75d631974635" containerName="payload-processing" containerID="cri-o://88351b4ba63fbcd8a0cd4410b2a7a85b5136d59051e9935f42dfa00fac578714" gracePeriod=25
Nov 03 21:41:08.989274 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:08.989188    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9" podUID="2625c6c8-87f6-43df-8607-15c0a2eb32c8" containerName="payload-pre-processing" containerID="cri-o://02303d41489a940491c735f79663f32d3e498a8fe47a0ababf8f5e24bc505b6c" gracePeriod=25
Nov 03 21:41:09.244680 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.244647    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2"
Nov 03 21:41:09.271003 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.270966    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57c5d5d3-f260-46e3-9ea8-75d631974635-kube-api-access-g4887\" (UniqueName: \"kubernetes.io/projected/57c5d5d3-f260-46e3-9ea8-75d631974635-kube-api-access-g4887\") pod \"57c5d5d3-f260-46e3-9ea8-75d631974635\" (UID: \"57c5d5d3-f260-46e3-9ea8-75d631974635\") "
Nov 03 21:41:09.271389 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.271365    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57c5d5d3-f260-46e3-9ea8-75d631974635-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57c5d5d3-f260-46e3-9ea8-75d631974635-plugins-config-volume\") pod \"57c5d5d3-f260-46e3-9ea8-75d631974635\" (UID: \"57c5d5d3-f260-46e3-9ea8-75d631974635\") "
Nov 03 21:41:09.271797 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.271772    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c5d5d3-f260-46e3-9ea8-75d631974635-plugins-config-volume" pod "57c5d5d3-f260-46e3-9ea8-75d631974635" (UID: "57c5d5d3-f260-46e3-9ea8-75d631974635"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:41:09.272005 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.271990    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57c5d5d3-f260-46e3-9ea8-75d631974635-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:09.275806 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.274369    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c5d5d3-f260-46e3-9ea8-75d631974635-kube-api-access-g4887" pod "57c5d5d3-f260-46e3-9ea8-75d631974635" (UID: "57c5d5d3-f260-46e3-9ea8-75d631974635"). InnerVolumeSpecName "kube-api-access-g4887". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:09.310316 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.310289    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9"
Nov 03 21:41:09.313957 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.313926    5053 generic.go:356] "Generic (PLEG): container finished" podID="57c5d5d3-f260-46e3-9ea8-75d631974635" containerID="88351b4ba63fbcd8a0cd4410b2a7a85b5136d59051e9935f42dfa00fac578714" exitCode=0
Nov 03 21:41:09.314106 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.313978    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2"
Nov 03 21:41:09.314106 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.314025    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2" event={"ID":"57c5d5d3-f260-46e3-9ea8-75d631974635","Type":"ContainerDied","Data":"88351b4ba63fbcd8a0cd4410b2a7a85b5136d59051e9935f42dfa00fac578714"}
Nov 03 21:41:09.314106 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.314103    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2" event={"ID":"57c5d5d3-f260-46e3-9ea8-75d631974635","Type":"ContainerDied","Data":"6f1a7ce02e39b35b1c85a7f41fe0499ed89e6745717de41f9afce97e6b0a05bd"}
Nov 03 21:41:09.314833 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.314137    5053 scope.go:122] "RemoveContainer" containerID="88351b4ba63fbcd8a0cd4410b2a7a85b5136d59051e9935f42dfa00fac578714"
Nov 03 21:41:09.316033 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.316010    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9"
Nov 03 21:41:09.316127 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.316069    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9" event={"ID":"2625c6c8-87f6-43df-8607-15c0a2eb32c8","Type":"ContainerDied","Data":"02303d41489a940491c735f79663f32d3e498a8fe47a0ababf8f5e24bc505b6c"}
Nov 03 21:41:09.316358 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.316333    5053 generic.go:356] "Generic (PLEG): container finished" podID="2625c6c8-87f6-43df-8607-15c0a2eb32c8" containerID="02303d41489a940491c735f79663f32d3e498a8fe47a0ababf8f5e24bc505b6c" exitCode=0
Nov 03 21:41:09.316500 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.316429    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9" event={"ID":"2625c6c8-87f6-43df-8607-15c0a2eb32c8","Type":"ContainerDied","Data":"7828bc2379c7750624ffa92bdc695e4ff4b424ec65e78baac82b1dd67f2f10ec"}
Nov 03 21:41:09.325816 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.325791    5053 scope.go:122] "RemoveContainer" containerID="88351b4ba63fbcd8a0cd4410b2a7a85b5136d59051e9935f42dfa00fac578714"
Nov 03 21:41:09.326150 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:41:09.326125    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88351b4ba63fbcd8a0cd4410b2a7a85b5136d59051e9935f42dfa00fac578714\": container with ID starting with 88351b4ba63fbcd8a0cd4410b2a7a85b5136d59051e9935f42dfa00fac578714 not found: ID does not exist" containerID="88351b4ba63fbcd8a0cd4410b2a7a85b5136d59051e9935f42dfa00fac578714"
Nov 03 21:41:09.326276 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.326164    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88351b4ba63fbcd8a0cd4410b2a7a85b5136d59051e9935f42dfa00fac578714"} err="failed to get container status \"88351b4ba63fbcd8a0cd4410b2a7a85b5136d59051e9935f42dfa00fac578714\": rpc error: code = NotFound desc = could not find container \"88351b4ba63fbcd8a0cd4410b2a7a85b5136d59051e9935f42dfa00fac578714\": container with ID starting with 88351b4ba63fbcd8a0cd4410b2a7a85b5136d59051e9935f42dfa00fac578714 not found: ID does not exist"
Nov 03 21:41:09.326276 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.326189    5053 scope.go:122] "RemoveContainer" containerID="02303d41489a940491c735f79663f32d3e498a8fe47a0ababf8f5e24bc505b6c"
Nov 03 21:41:09.337841 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.337814    5053 scope.go:122] "RemoveContainer" containerID="02303d41489a940491c735f79663f32d3e498a8fe47a0ababf8f5e24bc505b6c"
Nov 03 21:41:09.338154 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:41:09.338134    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02303d41489a940491c735f79663f32d3e498a8fe47a0ababf8f5e24bc505b6c\": container with ID starting with 02303d41489a940491c735f79663f32d3e498a8fe47a0ababf8f5e24bc505b6c not found: ID does not exist" containerID="02303d41489a940491c735f79663f32d3e498a8fe47a0ababf8f5e24bc505b6c"
Nov 03 21:41:09.338272 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.338163    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02303d41489a940491c735f79663f32d3e498a8fe47a0ababf8f5e24bc505b6c"} err="failed to get container status \"02303d41489a940491c735f79663f32d3e498a8fe47a0ababf8f5e24bc505b6c\": rpc error: code = NotFound desc = could not find container \"02303d41489a940491c735f79663f32d3e498a8fe47a0ababf8f5e24bc505b6c\": container with ID starting with 02303d41489a940491c735f79663f32d3e498a8fe47a0ababf8f5e24bc505b6c not found: ID does not exist"
Nov 03 21:41:09.372529 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.372492    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2625c6c8-87f6-43df-8607-15c0a2eb32c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2625c6c8-87f6-43df-8607-15c0a2eb32c8-plugins-config-volume\") pod \"2625c6c8-87f6-43df-8607-15c0a2eb32c8\" (UID: \"2625c6c8-87f6-43df-8607-15c0a2eb32c8\") "
Nov 03 21:41:09.372749 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.372611    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2625c6c8-87f6-43df-8607-15c0a2eb32c8-kube-api-access-kxrj2\" (UniqueName: \"kubernetes.io/projected/2625c6c8-87f6-43df-8607-15c0a2eb32c8-kube-api-access-kxrj2\") pod \"2625c6c8-87f6-43df-8607-15c0a2eb32c8\" (UID: \"2625c6c8-87f6-43df-8607-15c0a2eb32c8\") "
Nov 03 21:41:09.372947 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.372920    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2625c6c8-87f6-43df-8607-15c0a2eb32c8-plugins-config-volume" pod "2625c6c8-87f6-43df-8607-15c0a2eb32c8" (UID: "2625c6c8-87f6-43df-8607-15c0a2eb32c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:41:09.373502 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.373476    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4887\" (UniqueName: \"kubernetes.io/projected/57c5d5d3-f260-46e3-9ea8-75d631974635-kube-api-access-g4887\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:09.373619 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.373505    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2625c6c8-87f6-43df-8607-15c0a2eb32c8-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:09.375038 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.375016    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2625c6c8-87f6-43df-8607-15c0a2eb32c8-kube-api-access-kxrj2" pod "2625c6c8-87f6-43df-8607-15c0a2eb32c8" (UID: "2625c6c8-87f6-43df-8607-15c0a2eb32c8"). InnerVolumeSpecName "kube-api-access-kxrj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:09.469727 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.469635    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2"]
Nov 03 21:41:09.473672 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.473643    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c5025c-59d9b47886-9sbv2"]
Nov 03 21:41:09.474484 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.474462    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxrj2\" (UniqueName: \"kubernetes.io/projected/2625c6c8-87f6-43df-8607-15c0a2eb32c8-kube-api-access-kxrj2\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:09.668990 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.668957    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57c5d5d3-f260-46e3-9ea8-75d631974635" path="/var/lib/kubelet/pods/57c5d5d3-f260-46e3-9ea8-75d631974635/volumes"
Nov 03 21:41:09.755458 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.755365    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9"]
Nov 03 21:41:09.759443 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:09.759416    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5025c-8645f684bc-b9tz9"]
Nov 03 21:41:10.931030 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:41:10.930992    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:41:11.341711 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.341672    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:41:11.341875 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.341754    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k" podUID="b9a77c28-78e6-42ce-89cd-e9956e13bfc2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:41:11.641660 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.641627    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:41:11.692425 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.691937    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2625c6c8-87f6-43df-8607-15c0a2eb32c8" path="/var/lib/kubelet/pods/2625c6c8-87f6-43df-8607-15c0a2eb32c8/volumes"
Nov 03 21:41:11.715783 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.715081    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-credential-socket\") pod \"c43adf43-1729-4f0d-b6c7-3918c0fff604\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") "
Nov 03 21:41:11.715783 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.715250    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-data\") pod \"c43adf43-1729-4f0d-b6c7-3918c0fff604\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") "
Nov 03 21:41:11.715783 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.715287    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-token\" (UniqueName: \"kubernetes.io/projected/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-token\") pod \"c43adf43-1729-4f0d-b6c7-3918c0fff604\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") "
Nov 03 21:41:11.715783 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.715413    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c43adf43-1729-4f0d-b6c7-3918c0fff604-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c43adf43-1729-4f0d-b6c7-3918c0fff604-istiod-ca-cert\") pod \"c43adf43-1729-4f0d-b6c7-3918c0fff604\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") "
Nov 03 21:41:11.715783 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.715425    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-credential-socket" pod "c43adf43-1729-4f0d-b6c7-3918c0fff604" (UID: "c43adf43-1729-4f0d-b6c7-3918c0fff604"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.715783 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.715447    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-workload-certs\") pod \"c43adf43-1729-4f0d-b6c7-3918c0fff604\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") "
Nov 03 21:41:11.715783 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.715515    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c43adf43-1729-4f0d-b6c7-3918c0fff604-kube-api-access-vqn26\" (UniqueName: \"kubernetes.io/projected/c43adf43-1729-4f0d-b6c7-3918c0fff604-kube-api-access-vqn26\") pod \"c43adf43-1729-4f0d-b6c7-3918c0fff604\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") "
Nov 03 21:41:11.715783 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.715549    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-workload-socket\") pod \"c43adf43-1729-4f0d-b6c7-3918c0fff604\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") "
Nov 03 21:41:11.715783 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.715656    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-envoy\") pod \"c43adf43-1729-4f0d-b6c7-3918c0fff604\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") "
Nov 03 21:41:11.715783 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.715699    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-podinfo\") pod \"c43adf43-1729-4f0d-b6c7-3918c0fff604\" (UID: \"c43adf43-1729-4f0d-b6c7-3918c0fff604\") "
Nov 03 21:41:11.716944 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.716550    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.717395 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.717095    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-workload-socket" pod "c43adf43-1729-4f0d-b6c7-3918c0fff604" (UID: "c43adf43-1729-4f0d-b6c7-3918c0fff604"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.718434 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.718410    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-data" pod "c43adf43-1729-4f0d-b6c7-3918c0fff604" (UID: "c43adf43-1729-4f0d-b6c7-3918c0fff604"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.718714 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.718693    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c43adf43-1729-4f0d-b6c7-3918c0fff604-istiod-ca-cert" pod "c43adf43-1729-4f0d-b6c7-3918c0fff604" (UID: "c43adf43-1729-4f0d-b6c7-3918c0fff604"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:41:11.718983 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.718959    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-workload-certs" pod "c43adf43-1729-4f0d-b6c7-3918c0fff604" (UID: "c43adf43-1729-4f0d-b6c7-3918c0fff604"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.719119 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.719093    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-token" pod "c43adf43-1729-4f0d-b6c7-3918c0fff604" (UID: "c43adf43-1729-4f0d-b6c7-3918c0fff604"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:11.723304 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.723263    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c43adf43-1729-4f0d-b6c7-3918c0fff604-kube-api-access-vqn26" pod "c43adf43-1729-4f0d-b6c7-3918c0fff604" (UID: "c43adf43-1729-4f0d-b6c7-3918c0fff604"). InnerVolumeSpecName "kube-api-access-vqn26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:11.723529 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.723508    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-podinfo" pod "c43adf43-1729-4f0d-b6c7-3918c0fff604" (UID: "c43adf43-1729-4f0d-b6c7-3918c0fff604"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:41:11.723615 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.723508    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-envoy" pod "c43adf43-1729-4f0d-b6c7-3918c0fff604" (UID: "c43adf43-1729-4f0d-b6c7-3918c0fff604"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.756067 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.756033    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:41:11.817922 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.817889    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-workload-certs\") pod \"51d17433-dc86-491b-9276-080c84dfdc8e\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") "
Nov 03 21:41:11.818113 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.817973    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d17433-dc86-491b-9276-080c84dfdc8e-kube-api-access-clm7n\" (UniqueName: \"kubernetes.io/projected/51d17433-dc86-491b-9276-080c84dfdc8e-kube-api-access-clm7n\") pod \"51d17433-dc86-491b-9276-080c84dfdc8e\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") "
Nov 03 21:41:11.818113 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.818036    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-istio-envoy\") pod \"51d17433-dc86-491b-9276-080c84dfdc8e\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") "
Nov 03 21:41:11.818192 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.818113    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51d17433-dc86-491b-9276-080c84dfdc8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d17433-dc86-491b-9276-080c84dfdc8e-istiod-ca-cert\") pod \"51d17433-dc86-491b-9276-080c84dfdc8e\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") "
Nov 03 21:41:11.818192 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.818142    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-workload-socket\") pod \"51d17433-dc86-491b-9276-080c84dfdc8e\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") "
Nov 03 21:41:11.818362 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.818339    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51d17433-dc86-491b-9276-080c84dfdc8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d17433-dc86-491b-9276-080c84dfdc8e-istio-podinfo\") pod \"51d17433-dc86-491b-9276-080c84dfdc8e\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") "
Nov 03 21:41:11.818442 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.818390    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d17433-dc86-491b-9276-080c84dfdc8e-istio-token\" (UniqueName: \"kubernetes.io/projected/51d17433-dc86-491b-9276-080c84dfdc8e-istio-token\") pod \"51d17433-dc86-491b-9276-080c84dfdc8e\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") "
Nov 03 21:41:11.818493 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.818480    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-credential-socket\") pod \"51d17433-dc86-491b-9276-080c84dfdc8e\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") "
Nov 03 21:41:11.818569 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.818553    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-istio-data\") pod \"51d17433-dc86-491b-9276-080c84dfdc8e\" (UID: \"51d17433-dc86-491b-9276-080c84dfdc8e\") "
Nov 03 21:41:11.818853 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.818828    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-workload-certs" pod "51d17433-dc86-491b-9276-080c84dfdc8e" (UID: "51d17433-dc86-491b-9276-080c84dfdc8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.819000 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.818983    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-workload-socket" pod "51d17433-dc86-491b-9276-080c84dfdc8e" (UID: "51d17433-dc86-491b-9276-080c84dfdc8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.819393 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.819368    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.819651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.819398    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.819651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.819412    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.819651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.819424    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c43adf43-1729-4f0d-b6c7-3918c0fff604-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.819651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.819440    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.819651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.819454    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqn26\" (UniqueName: \"kubernetes.io/projected/c43adf43-1729-4f0d-b6c7-3918c0fff604-kube-api-access-vqn26\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.819651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.819468    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.819651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.819482    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.819651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.819495    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.819651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.819509    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c43adf43-1729-4f0d-b6c7-3918c0fff604-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.821338 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.821312    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-istio-envoy" pod "51d17433-dc86-491b-9276-080c84dfdc8e" (UID: "51d17433-dc86-491b-9276-080c84dfdc8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.821775 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.821603    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51d17433-dc86-491b-9276-080c84dfdc8e-istiod-ca-cert" pod "51d17433-dc86-491b-9276-080c84dfdc8e" (UID: "51d17433-dc86-491b-9276-080c84dfdc8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:41:11.821868 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.821841    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-credential-socket" pod "51d17433-dc86-491b-9276-080c84dfdc8e" (UID: "51d17433-dc86-491b-9276-080c84dfdc8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.823559 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.823222    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d17433-dc86-491b-9276-080c84dfdc8e-kube-api-access-clm7n" pod "51d17433-dc86-491b-9276-080c84dfdc8e" (UID: "51d17433-dc86-491b-9276-080c84dfdc8e"). InnerVolumeSpecName "kube-api-access-clm7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:11.823826 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.823796    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-istio-data" pod "51d17433-dc86-491b-9276-080c84dfdc8e" (UID: "51d17433-dc86-491b-9276-080c84dfdc8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.825463 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.825439    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d17433-dc86-491b-9276-080c84dfdc8e-istio-token" pod "51d17433-dc86-491b-9276-080c84dfdc8e" (UID: "51d17433-dc86-491b-9276-080c84dfdc8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:11.825616 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.825525    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51d17433-dc86-491b-9276-080c84dfdc8e-istio-podinfo" pod "51d17433-dc86-491b-9276-080c84dfdc8e" (UID: "51d17433-dc86-491b-9276-080c84dfdc8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:41:11.835783 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.835761    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:41:11.839888 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.839864    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:41:11.920268 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.920162    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istiod-ca-cert\") pod \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") "
Nov 03 21:41:11.920268 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.920208    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-token\" (UniqueName: \"kubernetes.io/projected/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-token\") pod \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") "
Nov 03 21:41:11.920485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.920288    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-envoy\") pod \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") "
Nov 03 21:41:11.920485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.920318    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-podinfo\") pod \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") "
Nov 03 21:41:11.920485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.920346    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-data\") pod \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") "
Nov 03 21:41:11.920485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.920397    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-credential-socket\") pod \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") "
Nov 03 21:41:11.920485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.920433    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-token\" (UniqueName: \"kubernetes.io/projected/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-token\") pod \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") "
Nov 03 21:41:11.920485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.920465    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-workload-certs\") pod \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") "
Nov 03 21:41:11.920850 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.920500    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-podinfo\") pod \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") "
Nov 03 21:41:11.920850 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.920559    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-credential-socket\") pod \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") "
Nov 03 21:41:11.920850 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.920567    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istiod-ca-cert" pod "b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa" (UID: "b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:41:11.920850 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.920622    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-data\") pod \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") "
Nov 03 21:41:11.920850 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.920661    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-envoy\") pod \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") "
Nov 03 21:41:11.920850 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.920695    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-credential-socket" pod "b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa" (UID: "b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.920850 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.920734    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-kube-api-access-2zvvf\" (UniqueName: \"kubernetes.io/projected/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-kube-api-access-2zvvf\") pod \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") "
Nov 03 21:41:11.920850 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.920779    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istiod-ca-cert\") pod \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") "
Nov 03 21:41:11.920850 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.920848    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-workload-certs\") pod \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") "
Nov 03 21:41:11.921543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.920889    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-workload-certs" pod "b9a77c28-78e6-42ce-89cd-e9956e13bfc2" (UID: "b9a77c28-78e6-42ce-89cd-e9956e13bfc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.921543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.920906    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-credential-socket" pod "b9a77c28-78e6-42ce-89cd-e9956e13bfc2" (UID: "b9a77c28-78e6-42ce-89cd-e9956e13bfc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.921543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.921089    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-workload-socket\") pod \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\" (UID: \"b9a77c28-78e6-42ce-89cd-e9956e13bfc2\") "
Nov 03 21:41:11.921543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.921310    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-workload-certs" pod "b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa" (UID: "b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.921543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.921335    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-workload-socket\") pod \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") "
Nov 03 21:41:11.921543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.921389    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-kube-api-access-5wwtl\" (UniqueName: \"kubernetes.io/projected/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-kube-api-access-5wwtl\") pod \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\" (UID: \"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa\") "
Nov 03 21:41:11.921543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.921477    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-data" pod "b9a77c28-78e6-42ce-89cd-e9956e13bfc2" (UID: "b9a77c28-78e6-42ce-89cd-e9956e13bfc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.921543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.921529    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-data" pod "b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa" (UID: "b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.921941 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.921827    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istiod-ca-cert" pod "b9a77c28-78e6-42ce-89cd-e9956e13bfc2" (UID: "b9a77c28-78e6-42ce-89cd-e9956e13bfc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:41:11.922940 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.922162    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d17433-dc86-491b-9276-080c84dfdc8e-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.922940 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.922192    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d17433-dc86-491b-9276-080c84dfdc8e-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.922940 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.922206    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.922940 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.922220    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.922940 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.922252    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.922940 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.922266    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.922940 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.922278    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.922940 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.922292    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.922940 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.922306    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.922940 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.922319    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.922940 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.922332    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.922940 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.922346    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clm7n\" (UniqueName: \"kubernetes.io/projected/51d17433-dc86-491b-9276-080c84dfdc8e-kube-api-access-clm7n\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.922940 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.922362    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.922940 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.922375    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d17433-dc86-491b-9276-080c84dfdc8e-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.922940 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.922389    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d17433-dc86-491b-9276-080c84dfdc8e-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.922940 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.922854    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-workload-socket" pod "b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa" (UID: "b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.922940 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.922906    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-workload-socket" pod "b9a77c28-78e6-42ce-89cd-e9956e13bfc2" (UID: "b9a77c28-78e6-42ce-89cd-e9956e13bfc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.923984 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.923716    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-podinfo" pod "b9a77c28-78e6-42ce-89cd-e9956e13bfc2" (UID: "b9a77c28-78e6-42ce-89cd-e9956e13bfc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:41:11.923984 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.923770    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-token" pod "b9a77c28-78e6-42ce-89cd-e9956e13bfc2" (UID: "b9a77c28-78e6-42ce-89cd-e9956e13bfc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:11.923984 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.923780    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-envoy" pod "b9a77c28-78e6-42ce-89cd-e9956e13bfc2" (UID: "b9a77c28-78e6-42ce-89cd-e9956e13bfc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.923984 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.923790    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-podinfo" pod "b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa" (UID: "b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:41:11.923984 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.923802    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-token" pod "b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa" (UID: "b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:11.924385 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.924124    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-envoy" pod "b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa" (UID: "b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.924926 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.924899    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-kube-api-access-2zvvf" pod "b9a77c28-78e6-42ce-89cd-e9956e13bfc2" (UID: "b9a77c28-78e6-42ce-89cd-e9956e13bfc2"). InnerVolumeSpecName "kube-api-access-2zvvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:11.925437 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:11.925419    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-kube-api-access-5wwtl" pod "b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa" (UID: "b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa"). InnerVolumeSpecName "kube-api-access-5wwtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:12.023979 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.023944    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.023979 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.023973    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.023979 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.023982    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.024617 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.023991    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zvvf\" (UniqueName: \"kubernetes.io/projected/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-kube-api-access-2zvvf\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.024617 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.024005    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.024617 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.024018    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.024617 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.024026    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wwtl\" (UniqueName: \"kubernetes.io/projected/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-kube-api-access-5wwtl\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.024617 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.024034    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.024617 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.024041    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a77c28-78e6-42ce-89cd-e9956e13bfc2-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.024617 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.024052    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.358962 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.358927    5053 generic.go:356] "Generic (PLEG): container finished" podID="b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa" containerID="c9ae10655d87ddcaea7236fc2fb980e11c5677bfb8a3addcc57f4b6edafe51b8" exitCode=0
Nov 03 21:41:12.359085 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.359068    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z" event={"ID":"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa","Type":"ContainerDied","Data":"c9ae10655d87ddcaea7236fc2fb980e11c5677bfb8a3addcc57f4b6edafe51b8"}
Nov 03 21:41:12.359142 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.359122    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z" event={"ID":"b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa","Type":"ContainerDied","Data":"c34e4c700c934b769f0412c09828262803e9c66486e16a21f8c8e7966f17c24a"}
Nov 03 21:41:12.359190 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.359155    5053 scope.go:122] "RemoveContainer" containerID="c9ae10655d87ddcaea7236fc2fb980e11c5677bfb8a3addcc57f4b6edafe51b8"
Nov 03 21:41:12.359373 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.359356    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"
Nov 03 21:41:12.377558 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.377519    5053 generic.go:356] "Generic (PLEG): container finished" podID="51d17433-dc86-491b-9276-080c84dfdc8e" containerID="b2d879b06a72cda2dd3d60517e935ff43844f9713ce2f1249ab29dcb6331a45b" exitCode=0
Nov 03 21:41:12.377719 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.377622    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd" event={"ID":"51d17433-dc86-491b-9276-080c84dfdc8e","Type":"ContainerDied","Data":"b2d879b06a72cda2dd3d60517e935ff43844f9713ce2f1249ab29dcb6331a45b"}
Nov 03 21:41:12.377719 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.377673    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd" event={"ID":"51d17433-dc86-491b-9276-080c84dfdc8e","Type":"ContainerDied","Data":"38376f3c5fd678a2115b066b4a967ee7c2a08b813511415fa5eb599364b7c995"}
Nov 03 21:41:12.377830 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.377764    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"
Nov 03 21:41:12.387191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.386976    5053 generic.go:356] "Generic (PLEG): container finished" podID="b9a77c28-78e6-42ce-89cd-e9956e13bfc2" containerID="565a74b41c0a9321ac8cdc1edbbadcdde952cfc70d0132c73a76748f5ac113fc" exitCode=0
Nov 03 21:41:12.387337 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.387220    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"
Nov 03 21:41:12.387413 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.387387    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k" event={"ID":"b9a77c28-78e6-42ce-89cd-e9956e13bfc2","Type":"ContainerDied","Data":"565a74b41c0a9321ac8cdc1edbbadcdde952cfc70d0132c73a76748f5ac113fc"}
Nov 03 21:41:12.387466 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.387440    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k" event={"ID":"b9a77c28-78e6-42ce-89cd-e9956e13bfc2","Type":"ContainerDied","Data":"ac23ccbe69a838105464b082ab06bf8b0f266aa751597fd62c7406b35e8170b6"}
Nov 03 21:41:12.391068 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.390005    5053 generic.go:356] "Generic (PLEG): container finished" podID="18feb2eb-b1a3-492c-8699-e507bf82b55f" containerID="40aac3ccec408709b293c389e67206c2382e7b8ab290222277764b399f8b8669" exitCode=0
Nov 03 21:41:12.391068 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.390178    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh" event={"ID":"18feb2eb-b1a3-492c-8699-e507bf82b55f","Type":"ContainerDied","Data":"40aac3ccec408709b293c389e67206c2382e7b8ab290222277764b399f8b8669"}
Nov 03 21:41:12.392583 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.392558    5053 generic.go:356] "Generic (PLEG): container finished" podID="c43adf43-1729-4f0d-b6c7-3918c0fff604" containerID="3b74edf2f27c8fddc02faf0dc29fa92b58b753482abcc5f5fc886630bd90214e" exitCode=0
Nov 03 21:41:12.392788 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.392768    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc" event={"ID":"c43adf43-1729-4f0d-b6c7-3918c0fff604","Type":"ContainerDied","Data":"3b74edf2f27c8fddc02faf0dc29fa92b58b753482abcc5f5fc886630bd90214e"}
Nov 03 21:41:12.392931 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.392914    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc" event={"ID":"c43adf43-1729-4f0d-b6c7-3918c0fff604","Type":"ContainerDied","Data":"b11439c52b6c2fd009ae82c47f9319cc7bf6b61c49bb690c8d47f9a140a43f37"}
Nov 03 21:41:12.395494 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.393156    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"
Nov 03 21:41:12.395919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.395796    5053 scope.go:122] "RemoveContainer" containerID="c9ae10655d87ddcaea7236fc2fb980e11c5677bfb8a3addcc57f4b6edafe51b8"
Nov 03 21:41:12.396403 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:41:12.396379    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9ae10655d87ddcaea7236fc2fb980e11c5677bfb8a3addcc57f4b6edafe51b8\": container with ID starting with c9ae10655d87ddcaea7236fc2fb980e11c5677bfb8a3addcc57f4b6edafe51b8 not found: ID does not exist" containerID="c9ae10655d87ddcaea7236fc2fb980e11c5677bfb8a3addcc57f4b6edafe51b8"
Nov 03 21:41:12.396472 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.396416    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9ae10655d87ddcaea7236fc2fb980e11c5677bfb8a3addcc57f4b6edafe51b8"} err="failed to get container status \"c9ae10655d87ddcaea7236fc2fb980e11c5677bfb8a3addcc57f4b6edafe51b8\": rpc error: code = NotFound desc = could not find container \"c9ae10655d87ddcaea7236fc2fb980e11c5677bfb8a3addcc57f4b6edafe51b8\": container with ID starting with c9ae10655d87ddcaea7236fc2fb980e11c5677bfb8a3addcc57f4b6edafe51b8 not found: ID does not exist"
Nov 03 21:41:12.396472 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.396437    5053 scope.go:122] "RemoveContainer" containerID="b2d879b06a72cda2dd3d60517e935ff43844f9713ce2f1249ab29dcb6331a45b"
Nov 03 21:41:12.397110 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.397089    5053 generic.go:356] "Generic (PLEG): container finished" podID="7b1d9743-441e-4e3a-a42b-c521bd99b7c3" containerID="c347183848d9e24f307629b3a84b05c9e6a9140a6a2bbfc77b5df89587e71004" exitCode=0
Nov 03 21:41:12.397206 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.397193    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6" event={"ID":"7b1d9743-441e-4e3a-a42b-c521bd99b7c3","Type":"ContainerDied","Data":"c347183848d9e24f307629b3a84b05c9e6a9140a6a2bbfc77b5df89587e71004"}
Nov 03 21:41:12.418493 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.413021    5053 scope.go:122] "RemoveContainer" containerID="77cc02f8698ec91ff89cc2168890c6970d6b9d597d41712c8ef537f3644d6f3f"
Nov 03 21:41:12.426050 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.426018    5053 scope.go:122] "RemoveContainer" containerID="b2d879b06a72cda2dd3d60517e935ff43844f9713ce2f1249ab29dcb6331a45b"
Nov 03 21:41:12.426676 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:41:12.426642    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d879b06a72cda2dd3d60517e935ff43844f9713ce2f1249ab29dcb6331a45b\": container with ID starting with b2d879b06a72cda2dd3d60517e935ff43844f9713ce2f1249ab29dcb6331a45b not found: ID does not exist" containerID="b2d879b06a72cda2dd3d60517e935ff43844f9713ce2f1249ab29dcb6331a45b"
Nov 03 21:41:12.426764 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.426687    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d879b06a72cda2dd3d60517e935ff43844f9713ce2f1249ab29dcb6331a45b"} err="failed to get container status \"b2d879b06a72cda2dd3d60517e935ff43844f9713ce2f1249ab29dcb6331a45b\": rpc error: code = NotFound desc = could not find container \"b2d879b06a72cda2dd3d60517e935ff43844f9713ce2f1249ab29dcb6331a45b\": container with ID starting with b2d879b06a72cda2dd3d60517e935ff43844f9713ce2f1249ab29dcb6331a45b not found: ID does not exist"
Nov 03 21:41:12.426764 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.426713    5053 scope.go:122] "RemoveContainer" containerID="77cc02f8698ec91ff89cc2168890c6970d6b9d597d41712c8ef537f3644d6f3f"
Nov 03 21:41:12.427219 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:41:12.427197    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77cc02f8698ec91ff89cc2168890c6970d6b9d597d41712c8ef537f3644d6f3f\": container with ID starting with 77cc02f8698ec91ff89cc2168890c6970d6b9d597d41712c8ef537f3644d6f3f not found: ID does not exist" containerID="77cc02f8698ec91ff89cc2168890c6970d6b9d597d41712c8ef537f3644d6f3f"
Nov 03 21:41:12.427671 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.427229    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77cc02f8698ec91ff89cc2168890c6970d6b9d597d41712c8ef537f3644d6f3f"} err="failed to get container status \"77cc02f8698ec91ff89cc2168890c6970d6b9d597d41712c8ef537f3644d6f3f\": rpc error: code = NotFound desc = could not find container \"77cc02f8698ec91ff89cc2168890c6970d6b9d597d41712c8ef537f3644d6f3f\": container with ID starting with 77cc02f8698ec91ff89cc2168890c6970d6b9d597d41712c8ef537f3644d6f3f not found: ID does not exist"
Nov 03 21:41:12.427671 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.427276    5053 scope.go:122] "RemoveContainer" containerID="565a74b41c0a9321ac8cdc1edbbadcdde952cfc70d0132c73a76748f5ac113fc"
Nov 03 21:41:12.441627 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.441439    5053 scope.go:122] "RemoveContainer" containerID="565a74b41c0a9321ac8cdc1edbbadcdde952cfc70d0132c73a76748f5ac113fc"
Nov 03 21:41:12.441840 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:41:12.441814    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"565a74b41c0a9321ac8cdc1edbbadcdde952cfc70d0132c73a76748f5ac113fc\": container with ID starting with 565a74b41c0a9321ac8cdc1edbbadcdde952cfc70d0132c73a76748f5ac113fc not found: ID does not exist" containerID="565a74b41c0a9321ac8cdc1edbbadcdde952cfc70d0132c73a76748f5ac113fc"
Nov 03 21:41:12.441911 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.441854    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"565a74b41c0a9321ac8cdc1edbbadcdde952cfc70d0132c73a76748f5ac113fc"} err="failed to get container status \"565a74b41c0a9321ac8cdc1edbbadcdde952cfc70d0132c73a76748f5ac113fc\": rpc error: code = NotFound desc = could not find container \"565a74b41c0a9321ac8cdc1edbbadcdde952cfc70d0132c73a76748f5ac113fc\": container with ID starting with 565a74b41c0a9321ac8cdc1edbbadcdde952cfc70d0132c73a76748f5ac113fc not found: ID does not exist"
Nov 03 21:41:12.441911 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.441880    5053 scope.go:122] "RemoveContainer" containerID="3b74edf2f27c8fddc02faf0dc29fa92b58b753482abcc5f5fc886630bd90214e"
Nov 03 21:41:12.495987 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.495960    5053 scope.go:122] "RemoveContainer" containerID="3b74edf2f27c8fddc02faf0dc29fa92b58b753482abcc5f5fc886630bd90214e"
Nov 03 21:41:12.497627 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:41:12.497592    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b74edf2f27c8fddc02faf0dc29fa92b58b753482abcc5f5fc886630bd90214e\": container with ID starting with 3b74edf2f27c8fddc02faf0dc29fa92b58b753482abcc5f5fc886630bd90214e not found: ID does not exist" containerID="3b74edf2f27c8fddc02faf0dc29fa92b58b753482abcc5f5fc886630bd90214e"
Nov 03 21:41:12.497790 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.497636    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b74edf2f27c8fddc02faf0dc29fa92b58b753482abcc5f5fc886630bd90214e"} err="failed to get container status \"3b74edf2f27c8fddc02faf0dc29fa92b58b753482abcc5f5fc886630bd90214e\": rpc error: code = NotFound desc = could not find container \"3b74edf2f27c8fddc02faf0dc29fa92b58b753482abcc5f5fc886630bd90214e\": container with ID starting with 3b74edf2f27c8fddc02faf0dc29fa92b58b753482abcc5f5fc886630bd90214e not found: ID does not exist"
Nov 03 21:41:12.499900 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.499875    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:41:12.531291 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.531261    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18feb2eb-b1a3-492c-8699-e507bf82b55f-kube-api-access-hx7q2\" (UniqueName: \"kubernetes.io/projected/18feb2eb-b1a3-492c-8699-e507bf82b55f-kube-api-access-hx7q2\") pod \"18feb2eb-b1a3-492c-8699-e507bf82b55f\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") "
Nov 03 21:41:12.531430 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.531312    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-podinfo\") pod \"18feb2eb-b1a3-492c-8699-e507bf82b55f\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") "
Nov 03 21:41:12.531430 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.531357    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-credential-socket\") pod \"18feb2eb-b1a3-492c-8699-e507bf82b55f\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") "
Nov 03 21:41:12.531430 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.531387    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-workload-socket\") pod \"18feb2eb-b1a3-492c-8699-e507bf82b55f\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") "
Nov 03 21:41:12.531610 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.531592    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-data\") pod \"18feb2eb-b1a3-492c-8699-e507bf82b55f\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") "
Nov 03 21:41:12.532428 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.531638    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-envoy\") pod \"18feb2eb-b1a3-492c-8699-e507bf82b55f\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") "
Nov 03 21:41:12.532428 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.531694    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-workload-socket" pod "18feb2eb-b1a3-492c-8699-e507bf82b55f" (UID: "18feb2eb-b1a3-492c-8699-e507bf82b55f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:12.532428 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.531704    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-credential-socket" pod "18feb2eb-b1a3-492c-8699-e507bf82b55f" (UID: "18feb2eb-b1a3-492c-8699-e507bf82b55f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:12.532428 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.531719    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18feb2eb-b1a3-492c-8699-e507bf82b55f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18feb2eb-b1a3-492c-8699-e507bf82b55f-istiod-ca-cert\") pod \"18feb2eb-b1a3-492c-8699-e507bf82b55f\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") "
Nov 03 21:41:12.532428 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.531762    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-workload-certs\") pod \"18feb2eb-b1a3-492c-8699-e507bf82b55f\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") "
Nov 03 21:41:12.532428 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.531835    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-data" pod "18feb2eb-b1a3-492c-8699-e507bf82b55f" (UID: "18feb2eb-b1a3-492c-8699-e507bf82b55f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:12.532428 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.531895    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-token\" (UniqueName: \"kubernetes.io/projected/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-token\") pod \"18feb2eb-b1a3-492c-8699-e507bf82b55f\" (UID: \"18feb2eb-b1a3-492c-8699-e507bf82b55f\") "
Nov 03 21:41:12.532428 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.532126    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-workload-certs" pod "18feb2eb-b1a3-492c-8699-e507bf82b55f" (UID: "18feb2eb-b1a3-492c-8699-e507bf82b55f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:12.532428 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.532141    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18feb2eb-b1a3-492c-8699-e507bf82b55f-istiod-ca-cert" pod "18feb2eb-b1a3-492c-8699-e507bf82b55f" (UID: "18feb2eb-b1a3-492c-8699-e507bf82b55f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:41:12.533103 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.532588    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.533103 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.532611    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.533103 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.532624    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.533103 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.532637    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18feb2eb-b1a3-492c-8699-e507bf82b55f-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.533103 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.532650    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.533992 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.533970    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-envoy" pod "18feb2eb-b1a3-492c-8699-e507bf82b55f" (UID: "18feb2eb-b1a3-492c-8699-e507bf82b55f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:12.534182 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.534159    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-podinfo" pod "18feb2eb-b1a3-492c-8699-e507bf82b55f" (UID: "18feb2eb-b1a3-492c-8699-e507bf82b55f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:41:12.534635 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.534610    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18feb2eb-b1a3-492c-8699-e507bf82b55f-kube-api-access-hx7q2" pod "18feb2eb-b1a3-492c-8699-e507bf82b55f" (UID: "18feb2eb-b1a3-492c-8699-e507bf82b55f"). InnerVolumeSpecName "kube-api-access-hx7q2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:12.535631 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.535609    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-token" pod "18feb2eb-b1a3-492c-8699-e507bf82b55f" (UID: "18feb2eb-b1a3-492c-8699-e507bf82b55f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:12.535848 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.535829    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"]
Nov 03 21:41:12.544579 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.544560    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:41:12.583924 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.583889    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-5nzhd"]
Nov 03 21:41:12.633551 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.633508    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-envoy\") pod \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") "
Nov 03 21:41:12.633727 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.633607    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-token\" (UniqueName: \"kubernetes.io/projected/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-token\") pod \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") "
Nov 03 21:41:12.633727 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.633668    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istiod-ca-cert\") pod \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") "
Nov 03 21:41:12.633816 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.633777    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-workload-socket\") pod \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") "
Nov 03 21:41:12.633872 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.633858    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-data\") pod \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") "
Nov 03 21:41:12.633975 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.633900    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-podinfo\") pod \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") "
Nov 03 21:41:12.633975 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.633968    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-kube-api-access-kfzk5\" (UniqueName: \"kubernetes.io/projected/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-kube-api-access-kfzk5\") pod \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") "
Nov 03 21:41:12.634084 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.634048    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-workload-certs\") pod \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") "
Nov 03 21:41:12.634144 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.634106    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-credential-socket\") pod \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\" (UID: \"7b1d9743-441e-4e3a-a42b-c521bd99b7c3\") "
Nov 03 21:41:12.634144 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.634103    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istiod-ca-cert" pod "7b1d9743-441e-4e3a-a42b-c521bd99b7c3" (UID: "7b1d9743-441e-4e3a-a42b-c521bd99b7c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:41:12.634144 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.634122    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-data" pod "7b1d9743-441e-4e3a-a42b-c521bd99b7c3" (UID: "7b1d9743-441e-4e3a-a42b-c521bd99b7c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:12.634350 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.634281    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-workload-socket" pod "7b1d9743-441e-4e3a-a42b-c521bd99b7c3" (UID: "7b1d9743-441e-4e3a-a42b-c521bd99b7c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:12.634408 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.634371    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-workload-certs" pod "7b1d9743-441e-4e3a-a42b-c521bd99b7c3" (UID: "7b1d9743-441e-4e3a-a42b-c521bd99b7c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:12.634704 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.634638    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-credential-socket" pod "7b1d9743-441e-4e3a-a42b-c521bd99b7c3" (UID: "7b1d9743-441e-4e3a-a42b-c521bd99b7c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:12.635155 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.635134    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.635317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.635161    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.635317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.635177    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx7q2\" (UniqueName: \"kubernetes.io/projected/18feb2eb-b1a3-492c-8699-e507bf82b55f-kube-api-access-hx7q2\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.635317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.635192    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.635317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.635205    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.635317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.635219    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.635317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.635270    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.635317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.635289    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.635317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.635302    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18feb2eb-b1a3-492c-8699-e507bf82b55f-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.635924 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.635902    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-envoy" pod "7b1d9743-441e-4e3a-a42b-c521bd99b7c3" (UID: "7b1d9743-441e-4e3a-a42b-c521bd99b7c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:12.636183 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.636165    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-podinfo" pod "7b1d9743-441e-4e3a-a42b-c521bd99b7c3" (UID: "7b1d9743-441e-4e3a-a42b-c521bd99b7c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:41:12.636275 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.636166    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-token" pod "7b1d9743-441e-4e3a-a42b-c521bd99b7c3" (UID: "7b1d9743-441e-4e3a-a42b-c521bd99b7c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:12.636362 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.636346    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-kube-api-access-kfzk5" pod "7b1d9743-441e-4e3a-a42b-c521bd99b7c3" (UID: "7b1d9743-441e-4e3a-a42b-c521bd99b7c3"). InnerVolumeSpecName "kube-api-access-kfzk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:12.693790 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.693765    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"]
Nov 03 21:41:12.697918 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.697888    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hxsrc"]
Nov 03 21:41:12.736548 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.736519    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.736548 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.736545    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfzk5\" (UniqueName: \"kubernetes.io/projected/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-kube-api-access-kfzk5\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.736548 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.736555    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.736763 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.736564    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b1d9743-441e-4e3a-a42b-c521bd99b7c3-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.808949 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.808896    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"]
Nov 03 21:41:12.812895 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.812864    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-dl67z"]
Nov 03 21:41:12.922458 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.922423    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"]
Nov 03 21:41:12.928869 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:12.928835    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-4pt5k"]
Nov 03 21:41:13.407654 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:13.407623    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"
Nov 03 21:41:13.407654 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:13.407638    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6" event={"ID":"7b1d9743-441e-4e3a-a42b-c521bd99b7c3","Type":"ContainerDied","Data":"7832b670a4fd109acd3aa64d8d7b70e96ccc9e9d5863d4b73b69e27ed4657424"}
Nov 03 21:41:13.408497 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:13.407734    5053 scope.go:122] "RemoveContainer" containerID="c347183848d9e24f307629b3a84b05c9e6a9140a6a2bbfc77b5df89587e71004"
Nov 03 21:41:13.418738 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:13.418715    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"
Nov 03 21:41:13.418887 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:13.418709    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh" event={"ID":"18feb2eb-b1a3-492c-8699-e507bf82b55f","Type":"ContainerDied","Data":"5ee9cce5ff1731a7645851b50aaf22d77748bb5d71f6a95b857f5b8e8b3d68d1"}
Nov 03 21:41:13.419711 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:13.419694    5053 scope.go:122] "RemoveContainer" containerID="40aac3ccec408709b293c389e67206c2382e7b8ab290222277764b399f8b8669"
Nov 03 21:41:13.572936 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:13.572854    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"]
Nov 03 21:41:13.576520 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:13.576496    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-6fxxh"]
Nov 03 21:41:13.669227 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:13.669149    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18feb2eb-b1a3-492c-8699-e507bf82b55f" path="/var/lib/kubelet/pods/18feb2eb-b1a3-492c-8699-e507bf82b55f/volumes"
Nov 03 21:41:13.669671 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:13.669655    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51d17433-dc86-491b-9276-080c84dfdc8e" path="/var/lib/kubelet/pods/51d17433-dc86-491b-9276-080c84dfdc8e/volumes"
Nov 03 21:41:13.670102 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:13.670088    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa" path="/var/lib/kubelet/pods/b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa/volumes"
Nov 03 21:41:13.670740 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:13.670723    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9a77c28-78e6-42ce-89cd-e9956e13bfc2" path="/var/lib/kubelet/pods/b9a77c28-78e6-42ce-89cd-e9956e13bfc2/volumes"
Nov 03 21:41:13.671150 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:13.671138    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c43adf43-1729-4f0d-b6c7-3918c0fff604" path="/var/lib/kubelet/pods/c43adf43-1729-4f0d-b6c7-3918c0fff604/volumes"
Nov 03 21:41:13.685689 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:13.685655    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"]
Nov 03 21:41:13.689601 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:13.689576    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-k5rz6"]
Nov 03 21:41:15.668804 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:15.668773    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b1d9743-441e-4e3a-a42b-c521bd99b7c3" path="/var/lib/kubelet/pods/7b1d9743-441e-4e3a-a42b-c521bd99b7c3/volumes"
Nov 03 21:41:18.183055 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.183025    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"]
Nov 03 21:41:18.184029 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184008    5053 cpu_manager.go:414] "containerMap: removing container" podUID="2625c6c8-87f6-43df-8607-15c0a2eb32c8" containerName="payload-pre-processing"
Nov 03 21:41:18.184102 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184033    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2625c6c8-87f6-43df-8607-15c0a2eb32c8" containerName="payload-pre-processing"
Nov 03 21:41:18.184102 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184057    5053 cpu_manager.go:414] "containerMap: removing container" podUID="2d8a1267-4c43-4f54-815d-79e78fef26a5" containerName="revoke-keys"
Nov 03 21:41:18.184102 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184064    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d8a1267-4c43-4f54-815d-79e78fef26a5" containerName="revoke-keys"
Nov 03 21:41:18.184102 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184080    5053 cpu_manager.go:414] "containerMap: removing container" podUID="b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa" containerName="istio-proxy"
Nov 03 21:41:18.184102 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184088    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa" containerName="istio-proxy"
Nov 03 21:41:18.184102 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184101    5053 cpu_manager.go:414] "containerMap: removing container" podUID="18feb2eb-b1a3-492c-8699-e507bf82b55f" containerName="istio-proxy"
Nov 03 21:41:18.184414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184108    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18feb2eb-b1a3-492c-8699-e507bf82b55f" containerName="istio-proxy"
Nov 03 21:41:18.184414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184121    5053 cpu_manager.go:414] "containerMap: removing container" podUID="e13ebd3d-b7a9-43d9-a490-4e6d37b573d7" containerName="payload-processing"
Nov 03 21:41:18.184414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184128    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13ebd3d-b7a9-43d9-a490-4e6d37b573d7" containerName="payload-processing"
Nov 03 21:41:18.184414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184143    5053 cpu_manager.go:414] "containerMap: removing container" podUID="7650dde2-539f-4535-b69f-8e90820be009" containerName="payload-pre-processing"
Nov 03 21:41:18.184414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184150    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7650dde2-539f-4535-b69f-8e90820be009" containerName="payload-pre-processing"
Nov 03 21:41:18.184414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184166    5053 cpu_manager.go:414] "containerMap: removing container" podUID="c43adf43-1729-4f0d-b6c7-3918c0fff604" containerName="istio-proxy"
Nov 03 21:41:18.184414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184172    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c43adf43-1729-4f0d-b6c7-3918c0fff604" containerName="istio-proxy"
Nov 03 21:41:18.184414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184185    5053 cpu_manager.go:414] "containerMap: removing container" podUID="b9a77c28-78e6-42ce-89cd-e9956e13bfc2" containerName="istio-proxy"
Nov 03 21:41:18.184414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184191    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a77c28-78e6-42ce-89cd-e9956e13bfc2" containerName="istio-proxy"
Nov 03 21:41:18.184414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184209    5053 cpu_manager.go:414] "containerMap: removing container" podUID="7b1d9743-441e-4e3a-a42b-c521bd99b7c3" containerName="istio-proxy"
Nov 03 21:41:18.184414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184216    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b1d9743-441e-4e3a-a42b-c521bd99b7c3" containerName="istio-proxy"
Nov 03 21:41:18.184414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184230    5053 cpu_manager.go:414] "containerMap: removing container" podUID="51d17433-dc86-491b-9276-080c84dfdc8e" containerName="istio-proxy"
Nov 03 21:41:18.184414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184253    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d17433-dc86-491b-9276-080c84dfdc8e" containerName="istio-proxy"
Nov 03 21:41:18.184414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184274    5053 cpu_manager.go:414] "containerMap: removing container" podUID="51d17433-dc86-491b-9276-080c84dfdc8e" containerName="istio-proxy"
Nov 03 21:41:18.184414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184281    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d17433-dc86-491b-9276-080c84dfdc8e" containerName="istio-proxy"
Nov 03 21:41:18.184414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184293    5053 cpu_manager.go:414] "containerMap: removing container" podUID="57c5d5d3-f260-46e3-9ea8-75d631974635" containerName="payload-processing"
Nov 03 21:41:18.184414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184299    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57c5d5d3-f260-46e3-9ea8-75d631974635" containerName="payload-processing"
Nov 03 21:41:18.185044 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184456    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d8a1267-4c43-4f54-815d-79e78fef26a5" containerName="revoke-keys"
Nov 03 21:41:18.185044 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184468    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7a2dddd-5328-44b5-a5ae-b82bfcd8d9aa" containerName="istio-proxy"
Nov 03 21:41:18.185044 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184479    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="18feb2eb-b1a3-492c-8699-e507bf82b55f" containerName="istio-proxy"
Nov 03 21:41:18.185044 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184486    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13ebd3d-b7a9-43d9-a490-4e6d37b573d7" containerName="payload-processing"
Nov 03 21:41:18.185044 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184496    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="7650dde2-539f-4535-b69f-8e90820be009" containerName="payload-pre-processing"
Nov 03 21:41:18.185044 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184506    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="c43adf43-1729-4f0d-b6c7-3918c0fff604" containerName="istio-proxy"
Nov 03 21:41:18.185044 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184515    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a77c28-78e6-42ce-89cd-e9956e13bfc2" containerName="istio-proxy"
Nov 03 21:41:18.185044 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184524    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b1d9743-441e-4e3a-a42b-c521bd99b7c3" containerName="istio-proxy"
Nov 03 21:41:18.185044 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184534    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d17433-dc86-491b-9276-080c84dfdc8e" containerName="istio-proxy"
Nov 03 21:41:18.185044 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184543    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d17433-dc86-491b-9276-080c84dfdc8e" containerName="istio-proxy"
Nov 03 21:41:18.185044 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184552    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="57c5d5d3-f260-46e3-9ea8-75d631974635" containerName="payload-processing"
Nov 03 21:41:18.185044 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.184566    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="2625c6c8-87f6-43df-8607-15c0a2eb32c8" containerName="payload-pre-processing"
Nov 03 21:41:18.189941 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.189922    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.268258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.242336    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-89ddad-openshift-default-dockercfg-d7qp9\""
Nov 03 21:41:18.327741 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.327707    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"]
Nov 03 21:41:18.341037 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.336157    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-workload-socket\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.341037 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.336311    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istiod-ca-cert\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.341037 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.336349    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-podinfo\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.341037 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.336495    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-envoy\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.341037 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.336589    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-data\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.341037 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.336621    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-token\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.341037 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.336644    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-workload-certs\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.341037 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.336675    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-credential-socket\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.341037 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.336695    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kmxh\" (UniqueName: \"kubernetes.io/projected/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-kube-api-access-5kmxh\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.438187 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.438098    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-data\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.438187 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.438150    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-token\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.438530 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.438508    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-workload-certs\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.438618 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.438558    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-credential-socket\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.438618 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.438571    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-data\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.438618 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.438583    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kmxh\" (UniqueName: \"kubernetes.io/projected/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-kube-api-access-5kmxh\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.438770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.438643    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-workload-socket\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.438827 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.438763    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istiod-ca-cert\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.438827 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.438800    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-credential-socket\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.438827 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.438800    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-podinfo\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.439010 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.438986    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-envoy\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.439339 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.439313    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-workload-certs\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.439795 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.439774    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istiod-ca-cert\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.440329 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.440283    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-workload-socket\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.445926 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.445902    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-envoy\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.448462 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.448435    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-podinfo\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.497894 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.497862    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-token\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.498084 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.498003    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kmxh\" (UniqueName: \"kubernetes.io/projected/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-kube-api-access-5kmxh\") pod \"e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") " pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.507100 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.507073    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:18.953516 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:41:18.953481    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d4ed21_ed2d_40ea_ad11_24a268d5ef0c.slice/crio-ef3ca9fd664cb2db27ea3fac16aeda0be223e84e5413cf854b15891f288d7bb8 WatchSource:0}: Error finding container ef3ca9fd664cb2db27ea3fac16aeda0be223e84e5413cf854b15891f288d7bb8: Status 404 returned error can't find the container with id ef3ca9fd664cb2db27ea3fac16aeda0be223e84e5413cf854b15891f288d7bb8
Nov 03 21:41:18.954069 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.954009    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"]
Nov 03 21:41:18.956958 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.956916    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:41:18.957036 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.957006    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:41:18.957069 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:18.957047    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:41:19.473308 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:19.473265    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt" event={"ID":"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c","Type":"ContainerStarted","Data":"ea2b8ae0e04cdd2189d3750064ed76bebcdc91dac800f1fa3721b6d91000fe17"}
Nov 03 21:41:19.473854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:19.473823    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt" event={"ID":"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c","Type":"ContainerStarted","Data":"ef3ca9fd664cb2db27ea3fac16aeda0be223e84e5413cf854b15891f288d7bb8"}
Nov 03 21:41:19.507983 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:19.507941    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:19.593229 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:19.593179    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt" podStartSLOduration=1.5931649060000002 podStartE2EDuration="1.593164906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:41:19.590497878 +0000 UTC m=+1594.084419809" watchObservedRunningTime="2026-11-03 21:41:19.593164906 +0000 UTC m=+1594.087086780"
Nov 03 21:41:20.513671 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:20.513643    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:21.496821 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:21.496789    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:21.498069 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:21.498040    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:41:28.232523 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:28.232485    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5"]
Nov 03 21:41:28.238298 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:28.238252    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5"
Nov 03 21:41:28.302321 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:28.300570    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-89ddad\""
Nov 03 21:41:28.311419 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:28.307887    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a7b45d4-526f-4568-8036-1a614c502fe5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5\" (UID: \"6a7b45d4-526f-4568-8036-1a614c502fe5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5"
Nov 03 21:41:28.311419 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:28.308011    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5558z\" (UniqueName: \"kubernetes.io/projected/6a7b45d4-526f-4568-8036-1a614c502fe5-kube-api-access-5558z\") pod \"payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5\" (UID: \"6a7b45d4-526f-4568-8036-1a614c502fe5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5"
Nov 03 21:41:28.367948 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:28.367723    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5"]
Nov 03 21:41:28.380427 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:28.372679    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn"]
Nov 03 21:41:28.391770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:28.384634    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn"
Nov 03 21:41:28.441168 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:28.441121    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c02684d1-aaec-4345-9ce7-4d910ed4fbe9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn\" (UID: \"c02684d1-aaec-4345-9ce7-4d910ed4fbe9\") " pod="openshift-ingress/payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn"
Nov 03 21:41:28.441409 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:28.441222    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5558z\" (UniqueName: \"kubernetes.io/projected/6a7b45d4-526f-4568-8036-1a614c502fe5-kube-api-access-5558z\") pod \"payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5\" (UID: \"6a7b45d4-526f-4568-8036-1a614c502fe5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5"
Nov 03 21:41:28.441484 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:28.441441    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7xn7\" (UniqueName: \"kubernetes.io/projected/c02684d1-aaec-4345-9ce7-4d910ed4fbe9-kube-api-access-g7xn7\") pod \"payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn\" (UID: \"c02684d1-aaec-4345-9ce7-4d910ed4fbe9\") " pod="openshift-ingress/payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn"
Nov 03 21:41:28.441535 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:28.441508    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a7b45d4-526f-4568-8036-1a614c502fe5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5\" (UID: \"6a7b45d4-526f-4568-8036-1a614c502fe5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5"
Nov 03 21:41:28.442907 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:28.442878    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a7b45d4-526f-4568-8036-1a614c502fe5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5\" (UID: \"6a7b45d4-526f-4568-8036-1a614c502fe5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5"
Nov 03 21:41:28.467141 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:28.464717    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-89ddad-dockercfg-psdzp\""
Nov 03 21:41:28.472805 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:28.472779    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn"]
Nov 03 21:41:28.500177 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:28.500095    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5558z\" (UniqueName: \"kubernetes.io/projected/6a7b45d4-526f-4568-8036-1a614c502fe5-kube-api-access-5558z\") pod \"payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5\" (UID: \"6a7b45d4-526f-4568-8036-1a614c502fe5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5"
Nov 03 21:41:28.542666 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:28.542629    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7xn7\" (UniqueName: \"kubernetes.io/projected/c02684d1-aaec-4345-9ce7-4d910ed4fbe9-kube-api-access-g7xn7\") pod \"payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn\" (UID: \"c02684d1-aaec-4345-9ce7-4d910ed4fbe9\") " pod="openshift-ingress/payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn"
Nov 03 21:41:28.542834 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:28.542756    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c02684d1-aaec-4345-9ce7-4d910ed4fbe9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn\" (UID: \"c02684d1-aaec-4345-9ce7-4d910ed4fbe9\") " pod="openshift-ingress/payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn"
Nov 03 21:41:28.543537 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:28.543510    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c02684d1-aaec-4345-9ce7-4d910ed4fbe9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn\" (UID: \"c02684d1-aaec-4345-9ce7-4d910ed4fbe9\") " pod="openshift-ingress/payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn"
Nov 03 21:41:28.559167 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:28.559138    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5"
Nov 03 21:41:28.599557 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:28.599524    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7xn7\" (UniqueName: \"kubernetes.io/projected/c02684d1-aaec-4345-9ce7-4d910ed4fbe9-kube-api-access-g7xn7\") pod \"payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn\" (UID: \"c02684d1-aaec-4345-9ce7-4d910ed4fbe9\") " pod="openshift-ingress/payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn"
Nov 03 21:41:28.786489 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:28.786397    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn"
Nov 03 21:41:28.950513 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:28.950482    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5"]
Nov 03 21:41:28.950724 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:41:28.950700    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7b45d4_526f_4568_8036_1a614c502fe5.slice/crio-e8041d572033bf1d072396b49b9198c0877efc98ba2aefec582a34a808005e3d WatchSource:0}: Error finding container e8041d572033bf1d072396b49b9198c0877efc98ba2aefec582a34a808005e3d: Status 404 returned error can't find the container with id e8041d572033bf1d072396b49b9198c0877efc98ba2aefec582a34a808005e3d
Nov 03 21:41:29.167614 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:29.167587    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn"]
Nov 03 21:41:29.168495 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:41:29.168471    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc02684d1_aaec_4345_9ce7_4d910ed4fbe9.slice/crio-8dce1aeaf3032b086d2d440828517f0bc650adf8147cc9d00c3e156f9ac2808c WatchSource:0}: Error finding container 8dce1aeaf3032b086d2d440828517f0bc650adf8147cc9d00c3e156f9ac2808c: Status 404 returned error can't find the container with id 8dce1aeaf3032b086d2d440828517f0bc650adf8147cc9d00c3e156f9ac2808c
Nov 03 21:41:29.569304 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:29.569265    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn" event={"ID":"c02684d1-aaec-4345-9ce7-4d910ed4fbe9","Type":"ContainerStarted","Data":"643b4b1c12b849298908c1fe70121643a007aa705e8b82c1f00f934b9126de1b"}
Nov 03 21:41:29.569304 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:29.569321    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn" event={"ID":"c02684d1-aaec-4345-9ce7-4d910ed4fbe9","Type":"ContainerStarted","Data":"8dce1aeaf3032b086d2d440828517f0bc650adf8147cc9d00c3e156f9ac2808c"}
Nov 03 21:41:29.569848 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:29.569415    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn"
Nov 03 21:41:29.570999 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:29.570978    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5" event={"ID":"6a7b45d4-526f-4568-8036-1a614c502fe5","Type":"ContainerStarted","Data":"600abcac298be4df0f73f08519a4a25dc3da7ec60e2644dafac732388642373c"}
Nov 03 21:41:29.571107 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:29.571015    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5" event={"ID":"6a7b45d4-526f-4568-8036-1a614c502fe5","Type":"ContainerStarted","Data":"e8041d572033bf1d072396b49b9198c0877efc98ba2aefec582a34a808005e3d"}
Nov 03 21:41:29.571107 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:29.571055    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5"
Nov 03 21:41:29.683270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:29.683208    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn" podStartSLOduration=1.683195132 podStartE2EDuration="1.683195132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:41:29.681864358 +0000 UTC m=+1604.175786235" watchObservedRunningTime="2026-11-03 21:41:29.683195132 +0000 UTC m=+1604.177117006"
Nov 03 21:41:29.797763 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:29.797694    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5" podStartSLOduration=1.797672688 podStartE2EDuration="1.797672688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:41:29.793841991 +0000 UTC m=+1604.287763882" watchObservedRunningTime="2026-11-03 21:41:29.797672688 +0000 UTC m=+1604.291594584"
Nov 03 21:41:37.375696 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:37.375661    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"]
Nov 03 21:41:37.376164 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:37.375848    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm" podUID="1743870f-2488-4b26-a01d-baa5b81ca42d" containerName="istio-proxy" containerID="cri-o://9fb0e93f07d7f62092ee7445bb273e3abe61635581d15cfd2e129e4b4da49692" gracePeriod=30
Nov 03 21:41:38.049640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:38.049604    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95613b9c-prwz2"]
Nov 03 21:41:38.096953 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:38.096921    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95613b9c-prwz2"]
Nov 03 21:41:39.668735 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:39.668698    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15905e54-b9dd-4554-9d1e-8511d866d387" path="/var/lib/kubelet/pods/15905e54-b9dd-4554-9d1e-8511d866d387/volumes"
Nov 03 21:41:40.581448 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:40.581416    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5"
Nov 03 21:41:40.581790 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:40.581488    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn"
Nov 03 21:41:40.943202 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:41:40.943169    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:41:42.662611 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.662582    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:41:42.700308 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.700218    5053 generic.go:356] "Generic (PLEG): container finished" podID="1743870f-2488-4b26-a01d-baa5b81ca42d" containerID="9fb0e93f07d7f62092ee7445bb273e3abe61635581d15cfd2e129e4b4da49692" exitCode=0
Nov 03 21:41:42.700308 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.700285    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"
Nov 03 21:41:42.700308 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.700305    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm" event={"ID":"1743870f-2488-4b26-a01d-baa5b81ca42d","Type":"ContainerDied","Data":"9fb0e93f07d7f62092ee7445bb273e3abe61635581d15cfd2e129e4b4da49692"}
Nov 03 21:41:42.700730 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.700358    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm" event={"ID":"1743870f-2488-4b26-a01d-baa5b81ca42d","Type":"ContainerDied","Data":"3b43f8ddd4fa7586b5d966dc9e3442d1ef2881b08e5900112b9ae5857af117e4"}
Nov 03 21:41:42.700730 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.700389    5053 scope.go:122] "RemoveContainer" containerID="9fb0e93f07d7f62092ee7445bb273e3abe61635581d15cfd2e129e4b4da49692"
Nov 03 21:41:42.702398 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.702377    5053 generic.go:356] "Generic (PLEG): container finished" podID="5a7feda9-6f61-4bf7-8561-05fec79b8b17" containerID="d817e5c652e1a0db2a2ea8d374f2d359ffb45317185a4a80e10c3efe9044d196" exitCode=0
Nov 03 21:41:42.702610 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.702509    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg" event={"ID":"5a7feda9-6f61-4bf7-8561-05fec79b8b17","Type":"ContainerDied","Data":"d817e5c652e1a0db2a2ea8d374f2d359ffb45317185a4a80e10c3efe9044d196"}
Nov 03 21:41:42.703109 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.703093    5053 scope.go:122] "RemoveContainer" containerID="d817e5c652e1a0db2a2ea8d374f2d359ffb45317185a4a80e10c3efe9044d196"
Nov 03 21:41:42.704744 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.704718    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:41:42.704807 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.704786    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:41:42.704842 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.704821    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:41:42.712392 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.712372    5053 scope.go:122] "RemoveContainer" containerID="9fb0e93f07d7f62092ee7445bb273e3abe61635581d15cfd2e129e4b4da49692"
Nov 03 21:41:42.712721 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:41:42.712697    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fb0e93f07d7f62092ee7445bb273e3abe61635581d15cfd2e129e4b4da49692\": container with ID starting with 9fb0e93f07d7f62092ee7445bb273e3abe61635581d15cfd2e129e4b4da49692 not found: ID does not exist" containerID="9fb0e93f07d7f62092ee7445bb273e3abe61635581d15cfd2e129e4b4da49692"
Nov 03 21:41:42.712781 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.712731    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fb0e93f07d7f62092ee7445bb273e3abe61635581d15cfd2e129e4b4da49692"} err="failed to get container status \"9fb0e93f07d7f62092ee7445bb273e3abe61635581d15cfd2e129e4b4da49692\": rpc error: code = NotFound desc = could not find container \"9fb0e93f07d7f62092ee7445bb273e3abe61635581d15cfd2e129e4b4da49692\": container with ID starting with 9fb0e93f07d7f62092ee7445bb273e3abe61635581d15cfd2e129e4b4da49692 not found: ID does not exist"
Nov 03 21:41:42.770702 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.770669    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-token\" (UniqueName: \"kubernetes.io/projected/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-token\") pod \"1743870f-2488-4b26-a01d-baa5b81ca42d\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") "
Nov 03 21:41:42.770853 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.770731    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-workload-socket\") pod \"1743870f-2488-4b26-a01d-baa5b81ca42d\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") "
Nov 03 21:41:42.770853 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.770845    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-envoy\") pod \"1743870f-2488-4b26-a01d-baa5b81ca42d\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") "
Nov 03 21:41:42.770975 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.770907    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-data\") pod \"1743870f-2488-4b26-a01d-baa5b81ca42d\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") "
Nov 03 21:41:42.770975 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.770968    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-workload-certs\") pod \"1743870f-2488-4b26-a01d-baa5b81ca42d\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") "
Nov 03 21:41:42.771075 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.771034    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1743870f-2488-4b26-a01d-baa5b81ca42d-kube-api-access-qn6gx\" (UniqueName: \"kubernetes.io/projected/1743870f-2488-4b26-a01d-baa5b81ca42d-kube-api-access-qn6gx\") pod \"1743870f-2488-4b26-a01d-baa5b81ca42d\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") "
Nov 03 21:41:42.771075 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.771060    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-podinfo\") pod \"1743870f-2488-4b26-a01d-baa5b81ca42d\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") "
Nov 03 21:41:42.771173 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.771073    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-workload-socket" pod "1743870f-2488-4b26-a01d-baa5b81ca42d" (UID: "1743870f-2488-4b26-a01d-baa5b81ca42d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:42.771173 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.771099    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1743870f-2488-4b26-a01d-baa5b81ca42d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1743870f-2488-4b26-a01d-baa5b81ca42d-istiod-ca-cert\") pod \"1743870f-2488-4b26-a01d-baa5b81ca42d\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") "
Nov 03 21:41:42.771173 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.771155    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-credential-socket\") pod \"1743870f-2488-4b26-a01d-baa5b81ca42d\" (UID: \"1743870f-2488-4b26-a01d-baa5b81ca42d\") "
Nov 03 21:41:42.771355 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.771275    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-workload-certs" pod "1743870f-2488-4b26-a01d-baa5b81ca42d" (UID: "1743870f-2488-4b26-a01d-baa5b81ca42d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:42.771355 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.771289    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-data" pod "1743870f-2488-4b26-a01d-baa5b81ca42d" (UID: "1743870f-2488-4b26-a01d-baa5b81ca42d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:42.772594 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.771775    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-credential-socket" pod "1743870f-2488-4b26-a01d-baa5b81ca42d" (UID: "1743870f-2488-4b26-a01d-baa5b81ca42d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:42.772594 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.772212    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:42.772594 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.772254    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:42.772594 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.772269    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:42.772594 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.772282    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:42.772594 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.772548    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1743870f-2488-4b26-a01d-baa5b81ca42d-istiod-ca-cert" pod "1743870f-2488-4b26-a01d-baa5b81ca42d" (UID: "1743870f-2488-4b26-a01d-baa5b81ca42d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:41:42.773326 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.773306    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-token" pod "1743870f-2488-4b26-a01d-baa5b81ca42d" (UID: "1743870f-2488-4b26-a01d-baa5b81ca42d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:42.773788 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.773771    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-podinfo" pod "1743870f-2488-4b26-a01d-baa5b81ca42d" (UID: "1743870f-2488-4b26-a01d-baa5b81ca42d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:41:42.773788 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.773777    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-envoy" pod "1743870f-2488-4b26-a01d-baa5b81ca42d" (UID: "1743870f-2488-4b26-a01d-baa5b81ca42d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:42.773874 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.773842    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1743870f-2488-4b26-a01d-baa5b81ca42d-kube-api-access-qn6gx" pod "1743870f-2488-4b26-a01d-baa5b81ca42d" (UID: "1743870f-2488-4b26-a01d-baa5b81ca42d"). InnerVolumeSpecName "kube-api-access-qn6gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:42.877160 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.877116    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn6gx\" (UniqueName: \"kubernetes.io/projected/1743870f-2488-4b26-a01d-baa5b81ca42d-kube-api-access-qn6gx\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:42.877160 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.877162    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:42.877508 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.877176    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1743870f-2488-4b26-a01d-baa5b81ca42d-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:42.877508 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.877189    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:42.877508 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:42.877205    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1743870f-2488-4b26-a01d-baa5b81ca42d-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:43.132707 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:43.132677    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"]
Nov 03 21:41:43.139794 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:43.139767    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-jrggm"]
Nov 03 21:41:43.368849 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:43.368766    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:41:43.669779 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:43.669706    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1743870f-2488-4b26-a01d-baa5b81ca42d" path="/var/lib/kubelet/pods/1743870f-2488-4b26-a01d-baa5b81ca42d/volumes"
Nov 03 21:41:43.736570 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:43.736539    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg" event={"ID":"5a7feda9-6f61-4bf7-8561-05fec79b8b17","Type":"ContainerStarted","Data":"f257c7973b51a056eaaf58dcea26fab1f4aeb08b7a85ef74f65e7b38147f7cde"}
Nov 03 21:41:44.638996 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:44.638961    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:41:44.640393 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:44.640368    5053 patch_prober.go:29] interesting pod/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 21:41:44.640502 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:44.640432    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg" podUID="5a7feda9-6f61-4bf7-8561-05fec79b8b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 21:41:45.046333 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:45.046300    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fad433-xg4nh"]
Nov 03 21:41:45.096499 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:45.096463    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fad433-xg4nh"]
Nov 03 21:41:45.639077 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:45.639047    5053 patch_prober.go:29] interesting pod/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 21:41:45.639354 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:45.639329    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg" podUID="5a7feda9-6f61-4bf7-8561-05fec79b8b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 21:41:45.670125 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:45.670092    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="489a5fc0-8d59-47de-8289-41120739ab55" path="/var/lib/kubelet/pods/489a5fc0-8d59-47de-8289-41120739ab55/volumes"
Nov 03 21:41:46.585282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:46.585248    5053 scope.go:122] "RemoveContainer" containerID="783333e23a74ba617043005d53804302f111ed3e2e0d24d9093fd83383d753a1"
Nov 03 21:41:46.595356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:46.595339    5053 scope.go:122] "RemoveContainer" containerID="727a7481af5a37e2b5b58c2fc3957fe512148424d98f07d712425baa1de17d2c"
Nov 03 21:41:46.635758 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:46.635628    5053 scope.go:122] "RemoveContainer" containerID="eeb78dcc6af67435cfd48df583a9c1696faf5a4a7ee01b6ae964e64534e6f1cb"
Nov 03 21:41:46.639156 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:46.639135    5053 patch_prober.go:29] interesting pod/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 21:41:46.639347 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:46.639327    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg" podUID="5a7feda9-6f61-4bf7-8561-05fec79b8b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 21:41:46.651575 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:46.651551    5053 scope.go:122] "RemoveContainer" containerID="6cd99b67dc382acd3ed6ed706f1feaeff873ea7b03b527036e91188b19ff42ac"
Nov 03 21:41:46.662622 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:46.662603    5053 scope.go:122] "RemoveContainer" containerID="2358a68f0ab7edfa29098ac5719e00114e2e1b6915ff6f0b7df349b906d25c35"
Nov 03 21:41:46.672618 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:46.672597    5053 scope.go:122] "RemoveContainer" containerID="649ca2e9fb512cfd3068dfcd0147c5b67d4622812ccb01582733e3a3781370ac"
Nov 03 21:41:46.682143 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:46.682114    5053 scope.go:122] "RemoveContainer" containerID="44d28cd51932c648c9fd7a50a7c1bbdcf632b5d73c07edfd9050a4012144163d"
Nov 03 21:41:47.639580 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:47.639538    5053 patch_prober.go:29] interesting pod/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 21:41:47.639986 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:47.639600    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg" podUID="5a7feda9-6f61-4bf7-8561-05fec79b8b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 21:41:48.639574 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:48.639537    5053 patch_prober.go:29] interesting pod/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 21:41:48.639770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:48.639601    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg" podUID="5a7feda9-6f61-4bf7-8561-05fec79b8b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 21:41:49.644191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:49.644162    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:41:49.644696 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:49.644627    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:41:49.645275 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:49.645257    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:41:55.047456 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:55.047413    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17c89b26-wcx94"]
Nov 03 21:41:55.097507 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:55.097471    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17c89b26-wcx94"]
Nov 03 21:41:55.103427 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:55.103399    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1db22669-bvpzd"]
Nov 03 21:41:55.106610 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:55.106591    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1db22669-bvpzd"]
Nov 03 21:41:55.668596 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:55.668560    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ad4db01-376c-40c2-966a-2f764b23fa5f" path="/var/lib/kubelet/pods/1ad4db01-376c-40c2-966a-2f764b23fa5f/volumes"
Nov 03 21:41:55.669126 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:41:55.669096    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4958759-1aab-42ad-b08b-263aa890a359" path="/var/lib/kubelet/pods/c4958759-1aab-42ad-b08b-263aa890a359/volumes"
Nov 03 21:42:10.928473 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:42:10.928431    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:42:14.065495 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:14.065456    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30f143-bqhhl"]
Nov 03 21:42:14.066363 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:14.066343    5053 cpu_manager.go:414] "containerMap: removing container" podUID="1743870f-2488-4b26-a01d-baa5b81ca42d" containerName="istio-proxy"
Nov 03 21:42:14.066478 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:14.066365    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1743870f-2488-4b26-a01d-baa5b81ca42d" containerName="istio-proxy"
Nov 03 21:42:14.066593 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:14.066578    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="1743870f-2488-4b26-a01d-baa5b81ca42d" containerName="istio-proxy"
Nov 03 21:42:14.070435 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:14.070413    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30f143-bqhhl"
Nov 03 21:42:14.127633 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:14.127599    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30f143-bqhhl"]
Nov 03 21:42:14.169556 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:14.169519    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:42:14.169556 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:14.169527    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:42:14.169780 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:14.169582    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:42:14.171850 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:14.171830    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:42:14.182373 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:14.182346    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsv48\" (UniqueName: \"kubernetes.io/projected/5f4f30b0-e837-46ff-8df0-ec486b0c873d-kube-api-access-tsv48\") pod \"maas-api-revoke-keys-e2e-api-a-30f143-bqhhl\" (UID: \"5f4f30b0-e837-46ff-8df0-ec486b0c873d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30f143-bqhhl"
Nov 03 21:42:14.182504 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:14.182416    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4f30b0-e837-46ff-8df0-ec486b0c873d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-30f143-bqhhl\" (UID: \"5f4f30b0-e837-46ff-8df0-ec486b0c873d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30f143-bqhhl"
Nov 03 21:42:14.182573 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:14.182553    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4f30b0-e837-46ff-8df0-ec486b0c873d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-30f143-bqhhl\" (UID: \"5f4f30b0-e837-46ff-8df0-ec486b0c873d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30f143-bqhhl"
Nov 03 21:42:14.283694 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:14.283650    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsv48\" (UniqueName: \"kubernetes.io/projected/5f4f30b0-e837-46ff-8df0-ec486b0c873d-kube-api-access-tsv48\") pod \"maas-api-revoke-keys-e2e-api-a-30f143-bqhhl\" (UID: \"5f4f30b0-e837-46ff-8df0-ec486b0c873d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30f143-bqhhl"
Nov 03 21:42:14.283897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:14.283745    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4f30b0-e837-46ff-8df0-ec486b0c873d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-30f143-bqhhl\" (UID: \"5f4f30b0-e837-46ff-8df0-ec486b0c873d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30f143-bqhhl"
Nov 03 21:42:14.283897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:14.283807    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4f30b0-e837-46ff-8df0-ec486b0c873d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-30f143-bqhhl\" (UID: \"5f4f30b0-e837-46ff-8df0-ec486b0c873d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30f143-bqhhl"
Nov 03 21:42:14.284433 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:14.284414    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4f30b0-e837-46ff-8df0-ec486b0c873d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-30f143-bqhhl\" (UID: \"5f4f30b0-e837-46ff-8df0-ec486b0c873d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30f143-bqhhl"
Nov 03 21:42:14.286282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:14.286261    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4f30b0-e837-46ff-8df0-ec486b0c873d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-30f143-bqhhl\" (UID: \"5f4f30b0-e837-46ff-8df0-ec486b0c873d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30f143-bqhhl"
Nov 03 21:42:14.340919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:14.340832    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsv48\" (UniqueName: \"kubernetes.io/projected/5f4f30b0-e837-46ff-8df0-ec486b0c873d-kube-api-access-tsv48\") pod \"maas-api-revoke-keys-e2e-api-a-30f143-bqhhl\" (UID: \"5f4f30b0-e837-46ff-8df0-ec486b0c873d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30f143-bqhhl"
Nov 03 21:42:14.383788 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:14.383754    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30f143-bqhhl"
Nov 03 21:42:14.772658 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:14.772614    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30f143-bqhhl"]
Nov 03 21:42:14.774865 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:42:14.774834    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f4f30b0_e837_46ff_8df0_ec486b0c873d.slice/crio-7c890826aadfb3ad225d7ac1201e56d112740a28e0203c327f8cd545268605ec WatchSource:0}: Error finding container 7c890826aadfb3ad225d7ac1201e56d112740a28e0203c327f8cd545268605ec: Status 404 returned error can't find the container with id 7c890826aadfb3ad225d7ac1201e56d112740a28e0203c327f8cd545268605ec
Nov 03 21:42:15.064787 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:15.064704    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30f143-bqhhl" event={"ID":"5f4f30b0-e837-46ff-8df0-ec486b0c873d","Type":"ContainerStarted","Data":"7c890826aadfb3ad225d7ac1201e56d112740a28e0203c327f8cd545268605ec"}
Nov 03 21:42:15.442733 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:15.442692    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"]
Nov 03 21:42:15.443176 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:15.442924    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g" podUID="190b08f2-f879-4a61-aee1-590afc3c4afc" containerName="istio-proxy" containerID="cri-o://b8ca2e535ba553875a6ed12efc532eda12d1a481def097fc7226e626b9517c7b" gracePeriod=30
Nov 03 21:42:16.072859 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:16.072830    5053 generic.go:356] "Generic (PLEG): container finished" podID="5f4f30b0-e837-46ff-8df0-ec486b0c873d" containerID="c75df8d88a30595c26b9358a9c195cf96704484d5a6643d43d992a5610986b1b" exitCode=0
Nov 03 21:42:16.073041 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:16.072924    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30f143-bqhhl" event={"ID":"5f4f30b0-e837-46ff-8df0-ec486b0c873d","Type":"ContainerDied","Data":"c75df8d88a30595c26b9358a9c195cf96704484d5a6643d43d992a5610986b1b"}
Nov 03 21:42:17.267137 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:17.265407    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30f143-bqhhl"
Nov 03 21:42:17.327121 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:17.327084    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f4f30b0-e837-46ff-8df0-ec486b0c873d-kube-api-access-tsv48\" (UniqueName: \"kubernetes.io/projected/5f4f30b0-e837-46ff-8df0-ec486b0c873d-kube-api-access-tsv48\") pod \"5f4f30b0-e837-46ff-8df0-ec486b0c873d\" (UID: \"5f4f30b0-e837-46ff-8df0-ec486b0c873d\") "
Nov 03 21:42:17.327336 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:17.327266    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f4f30b0-e837-46ff-8df0-ec486b0c873d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4f30b0-e837-46ff-8df0-ec486b0c873d-maas-api-xks-ca\") pod \"5f4f30b0-e837-46ff-8df0-ec486b0c873d\" (UID: \"5f4f30b0-e837-46ff-8df0-ec486b0c873d\") "
Nov 03 21:42:17.327448 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:17.327411    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f4f30b0-e837-46ff-8df0-ec486b0c873d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4f30b0-e837-46ff-8df0-ec486b0c873d-maas-api-openshift-ca\") pod \"5f4f30b0-e837-46ff-8df0-ec486b0c873d\" (UID: \"5f4f30b0-e837-46ff-8df0-ec486b0c873d\") "
Nov 03 21:42:17.328009 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:17.327969    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f4f30b0-e837-46ff-8df0-ec486b0c873d-maas-api-openshift-ca" pod "5f4f30b0-e837-46ff-8df0-ec486b0c873d" (UID: "5f4f30b0-e837-46ff-8df0-ec486b0c873d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:42:17.329499 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:17.329473    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f4f30b0-e837-46ff-8df0-ec486b0c873d-maas-api-xks-ca" pod "5f4f30b0-e837-46ff-8df0-ec486b0c873d" (UID: "5f4f30b0-e837-46ff-8df0-ec486b0c873d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:42:17.329648 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:17.329631    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f4f30b0-e837-46ff-8df0-ec486b0c873d-kube-api-access-tsv48" pod "5f4f30b0-e837-46ff-8df0-ec486b0c873d" (UID: "5f4f30b0-e837-46ff-8df0-ec486b0c873d"). InnerVolumeSpecName "kube-api-access-tsv48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:17.429943 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:17.429903    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsv48\" (UniqueName: \"kubernetes.io/projected/5f4f30b0-e837-46ff-8df0-ec486b0c873d-kube-api-access-tsv48\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:17.429943 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:17.429946    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4f30b0-e837-46ff-8df0-ec486b0c873d-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:17.430355 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:17.429961    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4f30b0-e837-46ff-8df0-ec486b0c873d-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:17.434325 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:17.434292    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"]
Nov 03 21:42:17.434682 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:17.434624    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx" podUID="74369a1d-5e9b-47a7-8a35-a6d2777e620b" containerName="istio-proxy" containerID="cri-o://9d25c2cdc544d9e07eb42edca7dcfdbf5c921ab29ce911d63a1cb3c6d6afd6e7" gracePeriod=30
Nov 03 21:42:17.440107 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:17.440080    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"]
Nov 03 21:42:17.440356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:17.440333    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k" podUID="720802a7-8dda-4dcc-8deb-d764518f49c1" containerName="istio-proxy" containerID="cri-o://a2dbc742beb4de19a46484ae484f3f33d51c4160142da69d83490929a3f21094" gracePeriod=30
Nov 03 21:42:17.443720 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:17.443692    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"]
Nov 03 21:42:17.443884 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:17.443865    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh" podUID="c9cd9ef8-5f64-442e-b66a-f6e399137667" containerName="istio-proxy" containerID="cri-o://af154710f0e2effa9373102b0cf0944216cb6509790064e21d20ccabb338fe28" gracePeriod=30
Nov 03 21:42:17.446894 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:17.446864    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"]
Nov 03 21:42:17.448063 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:17.447027    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6" podUID="12490bb7-88e5-449b-82c1-5a315a521d36" containerName="istio-proxy" containerID="cri-o://886e356ed56e3ce8bbcdbb526b5e3572b9f05d7a80920d883a2c5ab08643fa67" gracePeriod=30
Nov 03 21:42:18.089194 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:18.089167    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30f143-bqhhl"
Nov 03 21:42:18.089194 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:18.089185    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30f143-bqhhl" event={"ID":"5f4f30b0-e837-46ff-8df0-ec486b0c873d","Type":"ContainerDied","Data":"7c890826aadfb3ad225d7ac1201e56d112740a28e0203c327f8cd545268605ec"}
Nov 03 21:42:18.089451 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:18.089224    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c890826aadfb3ad225d7ac1201e56d112740a28e0203c327f8cd545268605ec"
Nov 03 21:42:19.394262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.390489    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"]
Nov 03 21:42:19.404366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.398040    5053 cpu_manager.go:414] "containerMap: removing container" podUID="5f4f30b0-e837-46ff-8df0-ec486b0c873d" containerName="revoke-keys"
Nov 03 21:42:19.404366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.398068    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f4f30b0-e837-46ff-8df0-ec486b0c873d" containerName="revoke-keys"
Nov 03 21:42:19.404366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.398763    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f4f30b0-e837-46ff-8df0-ec486b0c873d" containerName="revoke-keys"
Nov 03 21:42:19.426262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.425794    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.481223 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.480743    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:42:19.481223 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.480824    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6" podUID="12490bb7-88e5-449b-82c1-5a315a521d36" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:42:19.489890 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.489863    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"]
Nov 03 21:42:19.566768 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.566737    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.566952 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.566794    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.566952 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.566867    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.567084 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.566994    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.567084 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.567048    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.567154 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.567145    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91eb34d0-5de4-4a33-b596-0f69a14d4131-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.567188 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.567171    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.567319 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.567302    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.567402 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.567376    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drkzd\" (UniqueName: \"kubernetes.io/projected/91eb34d0-5de4-4a33-b596-0f69a14d4131-kube-api-access-drkzd\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.669251 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.669148    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91eb34d0-5de4-4a33-b596-0f69a14d4131-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.669251 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.669196    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.669515 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.669307    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.669515 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.669350    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drkzd\" (UniqueName: \"kubernetes.io/projected/91eb34d0-5de4-4a33-b596-0f69a14d4131-kube-api-access-drkzd\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.669515 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.669435    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.669515 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.669484    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.669515 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.669511    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.669754 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.669580    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.669754 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.669612    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.669754 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.669689    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.669907 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.669756    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.669957 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.669900    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.669957 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.669948    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.670070 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.669990    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91eb34d0-5de4-4a33-b596-0f69a14d4131-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.671960 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.671940    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.672164 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.672146    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.726002 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.725954    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.726168 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.726028    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drkzd\" (UniqueName: \"kubernetes.io/projected/91eb34d0-5de4-4a33-b596-0f69a14d4131-kube-api-access-drkzd\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:19.787554 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:19.787523    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:20.216820 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:20.216782    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"]
Nov 03 21:42:20.255786 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:42:20.255751    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91eb34d0_5de4_4a33_b596_0f69a14d4131.slice/crio-50bd617d4797db6c9fe6fe0f1481775bb68c8dfcac79c2752aaf204a9a7e941d WatchSource:0}: Error finding container 50bd617d4797db6c9fe6fe0f1481775bb68c8dfcac79c2752aaf204a9a7e941d: Status 404 returned error can't find the container with id 50bd617d4797db6c9fe6fe0f1481775bb68c8dfcac79c2752aaf204a9a7e941d
Nov 03 21:42:20.258590 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:20.258555    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:42:20.258700 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:20.258639    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:42:20.258700 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:20.258685    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:42:20.702855 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:20.702785    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29472d-qzbth"]
Nov 03 21:42:20.713179 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:20.713134    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29472d-qzbth"]
Nov 03 21:42:20.929079 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:20.920406    5053 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 03 21:42:20.929079 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:20.920471    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g" podUID="190b08f2-f879-4a61-aee1-590afc3c4afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 03 21:42:21.047050 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.047013    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9c1979a-sd6w9"]
Nov 03 21:42:21.087454 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.087417    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9c1979a-sd6w9"]
Nov 03 21:42:21.142503 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.142468    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9" event={"ID":"91eb34d0-5de4-4a33-b596-0f69a14d4131","Type":"ContainerStarted","Data":"c5a7d4810f87ba57f6d4dc5490b9f557478e2330f5cd4e7632ff8b06fccfeccf"}
Nov 03 21:42:21.142699 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.142562    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9" event={"ID":"91eb34d0-5de4-4a33-b596-0f69a14d4131","Type":"ContainerStarted","Data":"50bd617d4797db6c9fe6fe0f1481775bb68c8dfcac79c2752aaf204a9a7e941d"}
Nov 03 21:42:21.144890 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.144862    5053 generic.go:356] "Generic (PLEG): container finished" podID="190b08f2-f879-4a61-aee1-590afc3c4afc" containerID="b8ca2e535ba553875a6ed12efc532eda12d1a481def097fc7226e626b9517c7b" exitCode=0
Nov 03 21:42:21.145037 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.144895    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g" event={"ID":"190b08f2-f879-4a61-aee1-590afc3c4afc","Type":"ContainerDied","Data":"b8ca2e535ba553875a6ed12efc532eda12d1a481def097fc7226e626b9517c7b"}
Nov 03 21:42:21.188627 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.188564    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:42:21.262016 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.261963    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9" podStartSLOduration=2.261949182 podStartE2EDuration="2.261949182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:21.259172177 +0000 UTC m=+1655.753094063" watchObservedRunningTime="2026-11-03 21:42:21.261949182 +0000 UTC m=+1655.755871056"
Nov 03 21:42:21.285640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.285604    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:42:21.285829 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.285659    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx" podUID="74369a1d-5e9b-47a7-8a35-a6d2777e620b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:42:21.295860 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.295832    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-workload-socket\") pod \"190b08f2-f879-4a61-aee1-590afc3c4afc\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") "
Nov 03 21:42:21.296002 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.295874    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-podinfo\") pod \"190b08f2-f879-4a61-aee1-590afc3c4afc\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") "
Nov 03 21:42:21.296002 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.295912    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-token\" (UniqueName: \"kubernetes.io/projected/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-token\") pod \"190b08f2-f879-4a61-aee1-590afc3c4afc\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") "
Nov 03 21:42:21.296002 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.295928    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/190b08f2-f879-4a61-aee1-590afc3c4afc-kube-api-access-zbcwh\" (UniqueName: \"kubernetes.io/projected/190b08f2-f879-4a61-aee1-590afc3c4afc-kube-api-access-zbcwh\") pod \"190b08f2-f879-4a61-aee1-590afc3c4afc\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") "
Nov 03 21:42:21.296165 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.296004    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/190b08f2-f879-4a61-aee1-590afc3c4afc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/190b08f2-f879-4a61-aee1-590afc3c4afc-istiod-ca-cert\") pod \"190b08f2-f879-4a61-aee1-590afc3c4afc\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") "
Nov 03 21:42:21.296165 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.296070    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-credential-socket\") pod \"190b08f2-f879-4a61-aee1-590afc3c4afc\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") "
Nov 03 21:42:21.296165 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.296120    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-workload-certs\") pod \"190b08f2-f879-4a61-aee1-590afc3c4afc\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") "
Nov 03 21:42:21.296165 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.296144    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-workload-socket" pod "190b08f2-f879-4a61-aee1-590afc3c4afc" (UID: "190b08f2-f879-4a61-aee1-590afc3c4afc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:21.296414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.296154    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-data\") pod \"190b08f2-f879-4a61-aee1-590afc3c4afc\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") "
Nov 03 21:42:21.296414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.296333    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-envoy\") pod \"190b08f2-f879-4a61-aee1-590afc3c4afc\" (UID: \"190b08f2-f879-4a61-aee1-590afc3c4afc\") "
Nov 03 21:42:21.296414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.296384    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-data" pod "190b08f2-f879-4a61-aee1-590afc3c4afc" (UID: "190b08f2-f879-4a61-aee1-590afc3c4afc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:21.296414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.296400    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-credential-socket" pod "190b08f2-f879-4a61-aee1-590afc3c4afc" (UID: "190b08f2-f879-4a61-aee1-590afc3c4afc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:21.296600 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.296554    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-workload-certs" pod "190b08f2-f879-4a61-aee1-590afc3c4afc" (UID: "190b08f2-f879-4a61-aee1-590afc3c4afc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:21.296669 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.296635    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/190b08f2-f879-4a61-aee1-590afc3c4afc-istiod-ca-cert" pod "190b08f2-f879-4a61-aee1-590afc3c4afc" (UID: "190b08f2-f879-4a61-aee1-590afc3c4afc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:42:21.297172 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.297144    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:21.297172 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.297168    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/190b08f2-f879-4a61-aee1-590afc3c4afc-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:21.297387 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.297183    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:21.297387 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.297195    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:21.297387 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.297206    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:21.298932 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.298905    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-envoy" pod "190b08f2-f879-4a61-aee1-590afc3c4afc" (UID: "190b08f2-f879-4a61-aee1-590afc3c4afc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:21.299032 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.298936    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190b08f2-f879-4a61-aee1-590afc3c4afc-kube-api-access-zbcwh" pod "190b08f2-f879-4a61-aee1-590afc3c4afc" (UID: "190b08f2-f879-4a61-aee1-590afc3c4afc"). InnerVolumeSpecName "kube-api-access-zbcwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:21.299032 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.298965    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-token" pod "190b08f2-f879-4a61-aee1-590afc3c4afc" (UID: "190b08f2-f879-4a61-aee1-590afc3c4afc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:21.299032 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.298971    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-podinfo" pod "190b08f2-f879-4a61-aee1-590afc3c4afc" (UID: "190b08f2-f879-4a61-aee1-590afc3c4afc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:42:21.398274 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.398216    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:21.398274 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.398260    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:21.398274 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.398269    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/190b08f2-f879-4a61-aee1-590afc3c4afc-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:21.398274 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.398279    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbcwh\" (UniqueName: \"kubernetes.io/projected/190b08f2-f879-4a61-aee1-590afc3c4afc-kube-api-access-zbcwh\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:21.668843 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.668808    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0" path="/var/lib/kubelet/pods/6e8affbd-e66d-4ba0-a1e6-bd974f28d3b0/volumes"
Nov 03 21:42:21.669369 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.669355    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d192375-39a1-4b2a-8353-c04b9b4debfd" path="/var/lib/kubelet/pods/9d192375-39a1-4b2a-8353-c04b9b4debfd/volumes"
Nov 03 21:42:21.787890 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.787856    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:21.790029 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.790005    5053 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 03 21:42:21.790152 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:21.790058    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9" podUID="91eb34d0-5de4-4a33-b596-0f69a14d4131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 03 21:42:22.153347 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.153315    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"
Nov 03 21:42:22.153347 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.153324    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g" event={"ID":"190b08f2-f879-4a61-aee1-590afc3c4afc","Type":"ContainerDied","Data":"cae4c5b8a01232f190a02c9060d9293e78a6c146f01721518f2f859362faf054"}
Nov 03 21:42:22.153600 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.153390    5053 scope.go:122] "RemoveContainer" containerID="b8ca2e535ba553875a6ed12efc532eda12d1a481def097fc7226e626b9517c7b"
Nov 03 21:42:22.335044 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.335009    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"]
Nov 03 21:42:22.385345 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.385313    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-dnz7g"]
Nov 03 21:42:22.788784 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.788745    5053 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 03 21:42:22.789258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.788812    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9" podUID="91eb34d0-5de4-4a33-b596-0f69a14d4131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 03 21:42:22.858736 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.857528    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:42:22.949313 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.949275    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-podinfo\") pod \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") "
Nov 03 21:42:22.949467 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.949330    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-workload-certs\") pod \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") "
Nov 03 21:42:22.949467 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.949424    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-workload-socket\") pod \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") "
Nov 03 21:42:22.949571 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.949472    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74369a1d-5e9b-47a7-8a35-a6d2777e620b-kube-api-access-bbvl8\" (UniqueName: \"kubernetes.io/projected/74369a1d-5e9b-47a7-8a35-a6d2777e620b-kube-api-access-bbvl8\") pod \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") "
Nov 03 21:42:22.949571 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.949510    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-envoy\") pod \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") "
Nov 03 21:42:22.949671 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.949627    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-data\") pod \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") "
Nov 03 21:42:22.949724 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.949695    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-credential-socket\") pod \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") "
Nov 03 21:42:22.949807 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.949768    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istiod-ca-cert\") pod \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") "
Nov 03 21:42:22.949861 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.949841    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-token\" (UniqueName: \"kubernetes.io/projected/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-token\") pod \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\" (UID: \"74369a1d-5e9b-47a7-8a35-a6d2777e620b\") "
Nov 03 21:42:22.949912 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.949877    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-workload-socket" pod "74369a1d-5e9b-47a7-8a35-a6d2777e620b" (UID: "74369a1d-5e9b-47a7-8a35-a6d2777e620b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:22.950791 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.950766    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-data" pod "74369a1d-5e9b-47a7-8a35-a6d2777e620b" (UID: "74369a1d-5e9b-47a7-8a35-a6d2777e620b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:22.950901 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.950877    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:22.950901 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.950896    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:22.951574 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.951538    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-credential-socket" pod "74369a1d-5e9b-47a7-8a35-a6d2777e620b" (UID: "74369a1d-5e9b-47a7-8a35-a6d2777e620b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:22.951671 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.951634    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-workload-certs" pod "74369a1d-5e9b-47a7-8a35-a6d2777e620b" (UID: "74369a1d-5e9b-47a7-8a35-a6d2777e620b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:22.951872 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.951851    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istiod-ca-cert" pod "74369a1d-5e9b-47a7-8a35-a6d2777e620b" (UID: "74369a1d-5e9b-47a7-8a35-a6d2777e620b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:42:22.953812 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.953784    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-podinfo" pod "74369a1d-5e9b-47a7-8a35-a6d2777e620b" (UID: "74369a1d-5e9b-47a7-8a35-a6d2777e620b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:42:22.955447 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.955412    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74369a1d-5e9b-47a7-8a35-a6d2777e620b-kube-api-access-bbvl8" pod "74369a1d-5e9b-47a7-8a35-a6d2777e620b" (UID: "74369a1d-5e9b-47a7-8a35-a6d2777e620b"). InnerVolumeSpecName "kube-api-access-bbvl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:22.960262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.955798    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-token" pod "74369a1d-5e9b-47a7-8a35-a6d2777e620b" (UID: "74369a1d-5e9b-47a7-8a35-a6d2777e620b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:22.960262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.956202    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-envoy" pod "74369a1d-5e9b-47a7-8a35-a6d2777e620b" (UID: "74369a1d-5e9b-47a7-8a35-a6d2777e620b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:22.960262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.959510    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:42:22.963744 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:22.963724    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:42:23.048688 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.048662    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:42:23.051913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.051851    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-podinfo\") pod \"720802a7-8dda-4dcc-8deb-d764518f49c1\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") "
Nov 03 21:42:23.051913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.051910    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-token\" (UniqueName: \"kubernetes.io/projected/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-token\") pod \"720802a7-8dda-4dcc-8deb-d764518f49c1\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") "
Nov 03 21:42:23.052106 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.051951    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/720802a7-8dda-4dcc-8deb-d764518f49c1-kube-api-access-72vrj\" (UniqueName: \"kubernetes.io/projected/720802a7-8dda-4dcc-8deb-d764518f49c1-kube-api-access-72vrj\") pod \"720802a7-8dda-4dcc-8deb-d764518f49c1\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") "
Nov 03 21:42:23.052106 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.051988    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-workload-socket\") pod \"720802a7-8dda-4dcc-8deb-d764518f49c1\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") "
Nov 03 21:42:23.052106 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.052011    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/720802a7-8dda-4dcc-8deb-d764518f49c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720802a7-8dda-4dcc-8deb-d764518f49c1-istiod-ca-cert\") pod \"720802a7-8dda-4dcc-8deb-d764518f49c1\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") "
Nov 03 21:42:23.052106 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.052035    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-credential-socket\") pod \"720802a7-8dda-4dcc-8deb-d764518f49c1\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") "
Nov 03 21:42:23.052106 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.052051    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-envoy\") pod \"720802a7-8dda-4dcc-8deb-d764518f49c1\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") "
Nov 03 21:42:23.052401 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.052120    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-workload-certs\") pod \"720802a7-8dda-4dcc-8deb-d764518f49c1\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") "
Nov 03 21:42:23.052401 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.052150    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-data\") pod \"720802a7-8dda-4dcc-8deb-d764518f49c1\" (UID: \"720802a7-8dda-4dcc-8deb-d764518f49c1\") "
Nov 03 21:42:23.052531 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.052435    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/720802a7-8dda-4dcc-8deb-d764518f49c1-istiod-ca-cert" pod "720802a7-8dda-4dcc-8deb-d764518f49c1" (UID: "720802a7-8dda-4dcc-8deb-d764518f49c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:42:23.052888 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.052863    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-data" pod "720802a7-8dda-4dcc-8deb-d764518f49c1" (UID: "720802a7-8dda-4dcc-8deb-d764518f49c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:23.052888 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.052878    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-workload-certs" pod "720802a7-8dda-4dcc-8deb-d764518f49c1" (UID: "720802a7-8dda-4dcc-8deb-d764518f49c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:23.053029 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.052945    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-credential-socket" pod "720802a7-8dda-4dcc-8deb-d764518f49c1" (UID: "720802a7-8dda-4dcc-8deb-d764518f49c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:23.053083 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.053040    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.053083 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.053055    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-workload-socket" pod "720802a7-8dda-4dcc-8deb-d764518f49c1" (UID: "720802a7-8dda-4dcc-8deb-d764518f49c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:23.053083 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.053064    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.053259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.053102    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbvl8\" (UniqueName: \"kubernetes.io/projected/74369a1d-5e9b-47a7-8a35-a6d2777e620b-kube-api-access-bbvl8\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.053259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.053122    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.053259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.053137    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720802a7-8dda-4dcc-8deb-d764518f49c1-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.053259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.053152    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.053259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.053166    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74369a1d-5e9b-47a7-8a35-a6d2777e620b-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.053259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.053179    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.053259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.053193    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.053259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.053207    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.053259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.053219    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74369a1d-5e9b-47a7-8a35-a6d2777e620b-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.054759 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.054734    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-token" pod "720802a7-8dda-4dcc-8deb-d764518f49c1" (UID: "720802a7-8dda-4dcc-8deb-d764518f49c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:23.054875 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.054838    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-podinfo" pod "720802a7-8dda-4dcc-8deb-d764518f49c1" (UID: "720802a7-8dda-4dcc-8deb-d764518f49c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:42:23.055279 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.055218    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-envoy" pod "720802a7-8dda-4dcc-8deb-d764518f49c1" (UID: "720802a7-8dda-4dcc-8deb-d764518f49c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:23.055414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.055387    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/720802a7-8dda-4dcc-8deb-d764518f49c1-kube-api-access-72vrj" pod "720802a7-8dda-4dcc-8deb-d764518f49c1" (UID: "720802a7-8dda-4dcc-8deb-d764518f49c1"). InnerVolumeSpecName "kube-api-access-72vrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:23.155421 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.155385    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9cd9ef8-5f64-442e-b66a-f6e399137667-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9cd9ef8-5f64-442e-b66a-f6e399137667-istiod-ca-cert\") pod \"c9cd9ef8-5f64-442e-b66a-f6e399137667\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") "
Nov 03 21:42:23.155606 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.155461    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-envoy\") pod \"c9cd9ef8-5f64-442e-b66a-f6e399137667\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") "
Nov 03 21:42:23.155606 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.155496    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9cd9ef8-5f64-442e-b66a-f6e399137667-kube-api-access-4pdvt\" (UniqueName: \"kubernetes.io/projected/c9cd9ef8-5f64-442e-b66a-f6e399137667-kube-api-access-4pdvt\") pod \"c9cd9ef8-5f64-442e-b66a-f6e399137667\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") "
Nov 03 21:42:23.155606 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.155531    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-credential-socket\") pod \"12490bb7-88e5-449b-82c1-5a315a521d36\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") "
Nov 03 21:42:23.155606 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.155562    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-workload-socket\") pod \"c9cd9ef8-5f64-442e-b66a-f6e399137667\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") "
Nov 03 21:42:23.155606 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.155603    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-istio-envoy\") pod \"12490bb7-88e5-449b-82c1-5a315a521d36\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") "
Nov 03 21:42:23.155864 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.155659    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-istio-data\") pod \"12490bb7-88e5-449b-82c1-5a315a521d36\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") "
Nov 03 21:42:23.155864 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.155706    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12490bb7-88e5-449b-82c1-5a315a521d36-istio-token\" (UniqueName: \"kubernetes.io/projected/12490bb7-88e5-449b-82c1-5a315a521d36-istio-token\") pod \"12490bb7-88e5-449b-82c1-5a315a521d36\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") "
Nov 03 21:42:23.155864 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.155738    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12490bb7-88e5-449b-82c1-5a315a521d36-kube-api-access-8mjbz\" (UniqueName: \"kubernetes.io/projected/12490bb7-88e5-449b-82c1-5a315a521d36-kube-api-access-8mjbz\") pod \"12490bb7-88e5-449b-82c1-5a315a521d36\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") "
Nov 03 21:42:23.155864 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.155847    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12490bb7-88e5-449b-82c1-5a315a521d36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12490bb7-88e5-449b-82c1-5a315a521d36-istiod-ca-cert\") pod \"12490bb7-88e5-449b-82c1-5a315a521d36\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") "
Nov 03 21:42:23.156032 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.155919    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-workload-certs\") pod \"c9cd9ef8-5f64-442e-b66a-f6e399137667\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") "
Nov 03 21:42:23.156032 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.155963    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-workload-socket\") pod \"12490bb7-88e5-449b-82c1-5a315a521d36\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") "
Nov 03 21:42:23.156032 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.155989    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-workload-certs\") pod \"12490bb7-88e5-449b-82c1-5a315a521d36\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") "
Nov 03 21:42:23.156032 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.156020    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-podinfo\") pod \"c9cd9ef8-5f64-442e-b66a-f6e399137667\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") "
Nov 03 21:42:23.156149 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.156100    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12490bb7-88e5-449b-82c1-5a315a521d36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12490bb7-88e5-449b-82c1-5a315a521d36-istio-podinfo\") pod \"12490bb7-88e5-449b-82c1-5a315a521d36\" (UID: \"12490bb7-88e5-449b-82c1-5a315a521d36\") "
Nov 03 21:42:23.156149 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.156097    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-workload-socket" pod "c9cd9ef8-5f64-442e-b66a-f6e399137667" (UID: "c9cd9ef8-5f64-442e-b66a-f6e399137667"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:23.156149 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.156141    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-token\" (UniqueName: \"kubernetes.io/projected/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-token\") pod \"c9cd9ef8-5f64-442e-b66a-f6e399137667\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") "
Nov 03 21:42:23.156265 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.156157    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9cd9ef8-5f64-442e-b66a-f6e399137667-istiod-ca-cert" pod "c9cd9ef8-5f64-442e-b66a-f6e399137667" (UID: "c9cd9ef8-5f64-442e-b66a-f6e399137667"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:42:23.156265 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.156253    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-data\") pod \"c9cd9ef8-5f64-442e-b66a-f6e399137667\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") "
Nov 03 21:42:23.156336 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.156280    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-credential-socket\") pod \"c9cd9ef8-5f64-442e-b66a-f6e399137667\" (UID: \"c9cd9ef8-5f64-442e-b66a-f6e399137667\") "
Nov 03 21:42:23.156371 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.156329    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-workload-certs" pod "c9cd9ef8-5f64-442e-b66a-f6e399137667" (UID: "c9cd9ef8-5f64-442e-b66a-f6e399137667"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:23.156735 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.156711    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-istio-data" pod "12490bb7-88e5-449b-82c1-5a315a521d36" (UID: "12490bb7-88e5-449b-82c1-5a315a521d36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:23.156985 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.156933    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-credential-socket" pod "c9cd9ef8-5f64-442e-b66a-f6e399137667" (UID: "c9cd9ef8-5f64-442e-b66a-f6e399137667"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:23.157175 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.157140    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72vrj\" (UniqueName: \"kubernetes.io/projected/720802a7-8dda-4dcc-8deb-d764518f49c1-kube-api-access-72vrj\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.157175 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.157159    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.157317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.157174    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.157317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.157190    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.157317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.157204    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9cd9ef8-5f64-442e-b66a-f6e399137667-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.157317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.157218    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.157317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.157251    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.157317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.157264    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.157317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.157279    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.157317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.157299    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720802a7-8dda-4dcc-8deb-d764518f49c1-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.157764 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.157406    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-credential-socket" pod "12490bb7-88e5-449b-82c1-5a315a521d36" (UID: "12490bb7-88e5-449b-82c1-5a315a521d36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:23.157764 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.157510    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-workload-socket" pod "12490bb7-88e5-449b-82c1-5a315a521d36" (UID: "12490bb7-88e5-449b-82c1-5a315a521d36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:23.157764 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.157682    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-workload-certs" pod "12490bb7-88e5-449b-82c1-5a315a521d36" (UID: "12490bb7-88e5-449b-82c1-5a315a521d36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:23.158516 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.158488    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-envoy" pod "c9cd9ef8-5f64-442e-b66a-f6e399137667" (UID: "c9cd9ef8-5f64-442e-b66a-f6e399137667"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:23.159492 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.159466    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-data" pod "c9cd9ef8-5f64-442e-b66a-f6e399137667" (UID: "c9cd9ef8-5f64-442e-b66a-f6e399137667"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:23.160430 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.160401    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-podinfo" pod "c9cd9ef8-5f64-442e-b66a-f6e399137667" (UID: "c9cd9ef8-5f64-442e-b66a-f6e399137667"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:42:23.160682 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.160657    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12490bb7-88e5-449b-82c1-5a315a521d36-istiod-ca-cert" pod "12490bb7-88e5-449b-82c1-5a315a521d36" (UID: "12490bb7-88e5-449b-82c1-5a315a521d36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:42:23.160815 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.160782    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-istio-envoy" pod "12490bb7-88e5-449b-82c1-5a315a521d36" (UID: "12490bb7-88e5-449b-82c1-5a315a521d36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:23.161536 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.161514    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12490bb7-88e5-449b-82c1-5a315a521d36-istio-token" pod "12490bb7-88e5-449b-82c1-5a315a521d36" (UID: "12490bb7-88e5-449b-82c1-5a315a521d36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:23.161640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.161540    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9cd9ef8-5f64-442e-b66a-f6e399137667-kube-api-access-4pdvt" pod "c9cd9ef8-5f64-442e-b66a-f6e399137667" (UID: "c9cd9ef8-5f64-442e-b66a-f6e399137667"). InnerVolumeSpecName "kube-api-access-4pdvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:23.162445 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.162425    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12490bb7-88e5-449b-82c1-5a315a521d36-istio-podinfo" pod "12490bb7-88e5-449b-82c1-5a315a521d36" (UID: "12490bb7-88e5-449b-82c1-5a315a521d36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:42:23.162673 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.162653    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12490bb7-88e5-449b-82c1-5a315a521d36-kube-api-access-8mjbz" pod "12490bb7-88e5-449b-82c1-5a315a521d36" (UID: "12490bb7-88e5-449b-82c1-5a315a521d36"). InnerVolumeSpecName "kube-api-access-8mjbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:23.163799 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.163775    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-token" pod "c9cd9ef8-5f64-442e-b66a-f6e399137667" (UID: "c9cd9ef8-5f64-442e-b66a-f6e399137667"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:23.168523 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.168500    5053 generic.go:356] "Generic (PLEG): container finished" podID="74369a1d-5e9b-47a7-8a35-a6d2777e620b" containerID="9d25c2cdc544d9e07eb42edca7dcfdbf5c921ab29ce911d63a1cb3c6d6afd6e7" exitCode=0
Nov 03 21:42:23.168614 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.168561    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"
Nov 03 21:42:23.168614 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.168593    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx" event={"ID":"74369a1d-5e9b-47a7-8a35-a6d2777e620b","Type":"ContainerDied","Data":"9d25c2cdc544d9e07eb42edca7dcfdbf5c921ab29ce911d63a1cb3c6d6afd6e7"}
Nov 03 21:42:23.168735 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.168656    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx" event={"ID":"74369a1d-5e9b-47a7-8a35-a6d2777e620b","Type":"ContainerDied","Data":"5b4aa4ed5c70f1b20351478543dba05e3261a39af6fb6bf6b40064c0a42da064"}
Nov 03 21:42:23.168735 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.168687    5053 scope.go:122] "RemoveContainer" containerID="9d25c2cdc544d9e07eb42edca7dcfdbf5c921ab29ce911d63a1cb3c6d6afd6e7"
Nov 03 21:42:23.175439 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.175415    5053 generic.go:356] "Generic (PLEG): container finished" podID="12490bb7-88e5-449b-82c1-5a315a521d36" containerID="886e356ed56e3ce8bbcdbb526b5e3572b9f05d7a80920d883a2c5ab08643fa67" exitCode=0
Nov 03 21:42:23.175541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.175533    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"
Nov 03 21:42:23.176180 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.176098    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6" event={"ID":"12490bb7-88e5-449b-82c1-5a315a521d36","Type":"ContainerDied","Data":"886e356ed56e3ce8bbcdbb526b5e3572b9f05d7a80920d883a2c5ab08643fa67"}
Nov 03 21:42:23.176180 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.176149    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6" event={"ID":"12490bb7-88e5-449b-82c1-5a315a521d36","Type":"ContainerDied","Data":"2b2cbc8d360daea82d707ee15489391410a82ac74b8b52403a1a17e093788aa7"}
Nov 03 21:42:23.182272 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.181414    5053 generic.go:356] "Generic (PLEG): container finished" podID="720802a7-8dda-4dcc-8deb-d764518f49c1" containerID="a2dbc742beb4de19a46484ae484f3f33d51c4160142da69d83490929a3f21094" exitCode=0
Nov 03 21:42:23.182272 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.181464    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"
Nov 03 21:42:23.182272 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.181495    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k" event={"ID":"720802a7-8dda-4dcc-8deb-d764518f49c1","Type":"ContainerDied","Data":"a2dbc742beb4de19a46484ae484f3f33d51c4160142da69d83490929a3f21094"}
Nov 03 21:42:23.182272 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.181538    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k" event={"ID":"720802a7-8dda-4dcc-8deb-d764518f49c1","Type":"ContainerDied","Data":"970efc0b78f32654cd1c2a939f62fec18b9f258eb7f1a723c3ca16057ae97d60"}
Nov 03 21:42:23.183159 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.182835    5053 scope.go:122] "RemoveContainer" containerID="9d25c2cdc544d9e07eb42edca7dcfdbf5c921ab29ce911d63a1cb3c6d6afd6e7"
Nov 03 21:42:23.183552 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:42:23.183522    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d25c2cdc544d9e07eb42edca7dcfdbf5c921ab29ce911d63a1cb3c6d6afd6e7\": container with ID starting with 9d25c2cdc544d9e07eb42edca7dcfdbf5c921ab29ce911d63a1cb3c6d6afd6e7 not found: ID does not exist" containerID="9d25c2cdc544d9e07eb42edca7dcfdbf5c921ab29ce911d63a1cb3c6d6afd6e7"
Nov 03 21:42:23.183650 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.183560    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d25c2cdc544d9e07eb42edca7dcfdbf5c921ab29ce911d63a1cb3c6d6afd6e7"} err="failed to get container status \"9d25c2cdc544d9e07eb42edca7dcfdbf5c921ab29ce911d63a1cb3c6d6afd6e7\": rpc error: code = NotFound desc = could not find container \"9d25c2cdc544d9e07eb42edca7dcfdbf5c921ab29ce911d63a1cb3c6d6afd6e7\": container with ID starting with 9d25c2cdc544d9e07eb42edca7dcfdbf5c921ab29ce911d63a1cb3c6d6afd6e7 not found: ID does not exist"
Nov 03 21:42:23.183650 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.183581    5053 scope.go:122] "RemoveContainer" containerID="886e356ed56e3ce8bbcdbb526b5e3572b9f05d7a80920d883a2c5ab08643fa67"
Nov 03 21:42:23.186746 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.186704    5053 generic.go:356] "Generic (PLEG): container finished" podID="c9cd9ef8-5f64-442e-b66a-f6e399137667" containerID="af154710f0e2effa9373102b0cf0944216cb6509790064e21d20ccabb338fe28" exitCode=0
Nov 03 21:42:23.187783 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.187766    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"
Nov 03 21:42:23.189992 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.189852    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh" event={"ID":"c9cd9ef8-5f64-442e-b66a-f6e399137667","Type":"ContainerDied","Data":"af154710f0e2effa9373102b0cf0944216cb6509790064e21d20ccabb338fe28"}
Nov 03 21:42:23.189992 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.189897    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh" event={"ID":"c9cd9ef8-5f64-442e-b66a-f6e399137667","Type":"ContainerDied","Data":"e4f3277e61bf4186939fd507e58c393e249e5bd32f24f61dfb1bc3c087cab092"}
Nov 03 21:42:23.196417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.196404    5053 scope.go:122] "RemoveContainer" containerID="886e356ed56e3ce8bbcdbb526b5e3572b9f05d7a80920d883a2c5ab08643fa67"
Nov 03 21:42:23.196640 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:42:23.196621    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"886e356ed56e3ce8bbcdbb526b5e3572b9f05d7a80920d883a2c5ab08643fa67\": container with ID starting with 886e356ed56e3ce8bbcdbb526b5e3572b9f05d7a80920d883a2c5ab08643fa67 not found: ID does not exist" containerID="886e356ed56e3ce8bbcdbb526b5e3572b9f05d7a80920d883a2c5ab08643fa67"
Nov 03 21:42:23.196703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.196646    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"886e356ed56e3ce8bbcdbb526b5e3572b9f05d7a80920d883a2c5ab08643fa67"} err="failed to get container status \"886e356ed56e3ce8bbcdbb526b5e3572b9f05d7a80920d883a2c5ab08643fa67\": rpc error: code = NotFound desc = could not find container \"886e356ed56e3ce8bbcdbb526b5e3572b9f05d7a80920d883a2c5ab08643fa67\": container with ID starting with 886e356ed56e3ce8bbcdbb526b5e3572b9f05d7a80920d883a2c5ab08643fa67 not found: ID does not exist"
Nov 03 21:42:23.196703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.196664    5053 scope.go:122] "RemoveContainer" containerID="a2dbc742beb4de19a46484ae484f3f33d51c4160142da69d83490929a3f21094"
Nov 03 21:42:23.251495 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.245979    5053 scope.go:122] "RemoveContainer" containerID="a2dbc742beb4de19a46484ae484f3f33d51c4160142da69d83490929a3f21094"
Nov 03 21:42:23.251495 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:42:23.246371    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2dbc742beb4de19a46484ae484f3f33d51c4160142da69d83490929a3f21094\": container with ID starting with a2dbc742beb4de19a46484ae484f3f33d51c4160142da69d83490929a3f21094 not found: ID does not exist" containerID="a2dbc742beb4de19a46484ae484f3f33d51c4160142da69d83490929a3f21094"
Nov 03 21:42:23.251495 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.246399    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2dbc742beb4de19a46484ae484f3f33d51c4160142da69d83490929a3f21094"} err="failed to get container status \"a2dbc742beb4de19a46484ae484f3f33d51c4160142da69d83490929a3f21094\": rpc error: code = NotFound desc = could not find container \"a2dbc742beb4de19a46484ae484f3f33d51c4160142da69d83490929a3f21094\": container with ID starting with a2dbc742beb4de19a46484ae484f3f33d51c4160142da69d83490929a3f21094 not found: ID does not exist"
Nov 03 21:42:23.251495 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.246426    5053 scope.go:122] "RemoveContainer" containerID="af154710f0e2effa9373102b0cf0944216cb6509790064e21d20ccabb338fe28"
Nov 03 21:42:23.264640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.264617    5053 scope.go:122] "RemoveContainer" containerID="af154710f0e2effa9373102b0cf0944216cb6509790064e21d20ccabb338fe28"
Nov 03 21:42:23.265188 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:42:23.265118    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af154710f0e2effa9373102b0cf0944216cb6509790064e21d20ccabb338fe28\": container with ID starting with af154710f0e2effa9373102b0cf0944216cb6509790064e21d20ccabb338fe28 not found: ID does not exist" containerID="af154710f0e2effa9373102b0cf0944216cb6509790064e21d20ccabb338fe28"
Nov 03 21:42:23.265188 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.265153    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af154710f0e2effa9373102b0cf0944216cb6509790064e21d20ccabb338fe28"} err="failed to get container status \"af154710f0e2effa9373102b0cf0944216cb6509790064e21d20ccabb338fe28\": rpc error: code = NotFound desc = could not find container \"af154710f0e2effa9373102b0cf0944216cb6509790064e21d20ccabb338fe28\": container with ID starting with af154710f0e2effa9373102b0cf0944216cb6509790064e21d20ccabb338fe28 not found: ID does not exist"
Nov 03 21:42:23.266380 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.265803    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.266380 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.265833    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pdvt\" (UniqueName: \"kubernetes.io/projected/c9cd9ef8-5f64-442e-b66a-f6e399137667-kube-api-access-4pdvt\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.266380 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.265851    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.266380 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.265866    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.266380 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.265881    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12490bb7-88e5-449b-82c1-5a315a521d36-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.266380 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.265894    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mjbz\" (UniqueName: \"kubernetes.io/projected/12490bb7-88e5-449b-82c1-5a315a521d36-kube-api-access-8mjbz\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.266380 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.265906    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12490bb7-88e5-449b-82c1-5a315a521d36-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.266380 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.265918    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.266380 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.265932    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12490bb7-88e5-449b-82c1-5a315a521d36-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.266380 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.265946    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.266380 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.265960    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12490bb7-88e5-449b-82c1-5a315a521d36-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.266380 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.265985    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.266380 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.266008    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9cd9ef8-5f64-442e-b66a-f6e399137667-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.443640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.443618    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"]
Nov 03 21:42:23.446846 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.446824    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh"]
Nov 03 21:42:23.556256 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.554570    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"]
Nov 03 21:42:23.566523 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.566115    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-l55t6"]
Nov 03 21:42:23.668056 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.668014    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12490bb7-88e5-449b-82c1-5a315a521d36" path="/var/lib/kubelet/pods/12490bb7-88e5-449b-82c1-5a315a521d36/volumes"
Nov 03 21:42:23.668640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.668618    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="190b08f2-f879-4a61-aee1-590afc3c4afc" path="/var/lib/kubelet/pods/190b08f2-f879-4a61-aee1-590afc3c4afc/volumes"
Nov 03 21:42:23.669133 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.669111    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9cd9ef8-5f64-442e-b66a-f6e399137667" path="/var/lib/kubelet/pods/c9cd9ef8-5f64-442e-b66a-f6e399137667/volumes"
Nov 03 21:42:23.684177 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.684149    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"]
Nov 03 21:42:23.689356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.689341    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-gqf6k"]
Nov 03 21:42:23.788294 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.788175    5053 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 03 21:42:23.788294 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.788268    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9" podUID="91eb34d0-5de4-4a33-b596-0f69a14d4131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 03 21:42:23.792759 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.792272    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"]
Nov 03 21:42:23.795687 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.795660    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tqkbx"]
Nov 03 21:42:23.878530 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.878494    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:42:23.878709 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:23.878551    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-qjthh" podUID="c9cd9ef8-5f64-442e-b66a-f6e399137667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:42:24.788735 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:24.788702    5053 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 03 21:42:24.788900 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:24.788751    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9" podUID="91eb34d0-5de4-4a33-b596-0f69a14d4131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 03 21:42:25.176229 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.176145    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"]
Nov 03 21:42:25.176794 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.176779    5053 cpu_manager.go:414] "containerMap: removing container" podUID="12490bb7-88e5-449b-82c1-5a315a521d36" containerName="istio-proxy"
Nov 03 21:42:25.176839 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.176795    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12490bb7-88e5-449b-82c1-5a315a521d36" containerName="istio-proxy"
Nov 03 21:42:25.176839 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.176808    5053 cpu_manager.go:414] "containerMap: removing container" podUID="c9cd9ef8-5f64-442e-b66a-f6e399137667" containerName="istio-proxy"
Nov 03 21:42:25.176839 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.176812    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9cd9ef8-5f64-442e-b66a-f6e399137667" containerName="istio-proxy"
Nov 03 21:42:25.176839 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.176826    5053 cpu_manager.go:414] "containerMap: removing container" podUID="190b08f2-f879-4a61-aee1-590afc3c4afc" containerName="istio-proxy"
Nov 03 21:42:25.176839 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.176831    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="190b08f2-f879-4a61-aee1-590afc3c4afc" containerName="istio-proxy"
Nov 03 21:42:25.176975 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.176843    5053 cpu_manager.go:414] "containerMap: removing container" podUID="74369a1d-5e9b-47a7-8a35-a6d2777e620b" containerName="istio-proxy"
Nov 03 21:42:25.176975 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.176847    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74369a1d-5e9b-47a7-8a35-a6d2777e620b" containerName="istio-proxy"
Nov 03 21:42:25.176975 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.176879    5053 cpu_manager.go:414] "containerMap: removing container" podUID="720802a7-8dda-4dcc-8deb-d764518f49c1" containerName="istio-proxy"
Nov 03 21:42:25.176975 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.176883    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="720802a7-8dda-4dcc-8deb-d764518f49c1" containerName="istio-proxy"
Nov 03 21:42:25.176975 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.176971    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="190b08f2-f879-4a61-aee1-590afc3c4afc" containerName="istio-proxy"
Nov 03 21:42:25.177109 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.176978    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="74369a1d-5e9b-47a7-8a35-a6d2777e620b" containerName="istio-proxy"
Nov 03 21:42:25.177109 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.176992    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="720802a7-8dda-4dcc-8deb-d764518f49c1" containerName="istio-proxy"
Nov 03 21:42:25.177109 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.177000    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="12490bb7-88e5-449b-82c1-5a315a521d36" containerName="istio-proxy"
Nov 03 21:42:25.177109 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.177006    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9cd9ef8-5f64-442e-b66a-f6e399137667" containerName="istio-proxy"
Nov 03 21:42:25.182283 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.182266    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"
Nov 03 21:42:25.189991 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.189964    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-openshift-service-ca\") pod \"maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9\" (UID: \"9bf836a1-d2b2-4f9b-bca1-92a40c71811b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"
Nov 03 21:42:25.190173 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.190154    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-maas-api-tls\") pod \"maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9\" (UID: \"9bf836a1-d2b2-4f9b-bca1-92a40c71811b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"
Nov 03 21:42:25.190461 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.190440    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-metrics-certs\") pod \"maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9\" (UID: \"9bf836a1-d2b2-4f9b-bca1-92a40c71811b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"
Nov 03 21:42:25.190530 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.190481    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zcbh\" (UniqueName: \"kubernetes.io/projected/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-kube-api-access-2zcbh\") pod \"maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9\" (UID: \"9bf836a1-d2b2-4f9b-bca1-92a40c71811b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"
Nov 03 21:42:25.286008 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.285970    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-89ddad-serving-cert\""
Nov 03 21:42:25.286008 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.286001    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:42:25.286326 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.286013    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 21:42:25.286326 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.286010    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:42:25.286326 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.286060    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nbmd6\""
Nov 03 21:42:25.289579 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.289551    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"]
Nov 03 21:42:25.291931 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.291897    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-metrics-certs\") pod \"maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9\" (UID: \"9bf836a1-d2b2-4f9b-bca1-92a40c71811b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"
Nov 03 21:42:25.291931 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.291932    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zcbh\" (UniqueName: \"kubernetes.io/projected/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-kube-api-access-2zcbh\") pod \"maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9\" (UID: \"9bf836a1-d2b2-4f9b-bca1-92a40c71811b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"
Nov 03 21:42:25.292125 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.291987    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-openshift-service-ca\") pod \"maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9\" (UID: \"9bf836a1-d2b2-4f9b-bca1-92a40c71811b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"
Nov 03 21:42:25.292125 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.292064    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-maas-api-tls\") pod \"maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9\" (UID: \"9bf836a1-d2b2-4f9b-bca1-92a40c71811b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"
Nov 03 21:42:25.292277 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:42:25.292223    5053 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-serving-cert: secret "maas-api-e2e-api-b-89ddad-serving-cert" not found
Nov 03 21:42:25.292340 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:42:25.292311    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-maas-api-tls podName:9bf836a1-d2b2-4f9b-bca1-92a40c71811b nodeName:}" failed. No retries permitted until 2026-11-03 21:42:25.792292232 +0000 UTC m=+1660.286214102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-maas-api-tls") pod "maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9" (UID: "9bf836a1-d2b2-4f9b-bca1-92a40c71811b") : secret "maas-api-e2e-api-b-89ddad-serving-cert" not found
Nov 03 21:42:25.292701 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.292681    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-openshift-service-ca\") pod \"maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9\" (UID: \"9bf836a1-d2b2-4f9b-bca1-92a40c71811b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"
Nov 03 21:42:25.294533 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.294511    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-metrics-certs\") pod \"maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9\" (UID: \"9bf836a1-d2b2-4f9b-bca1-92a40c71811b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"
Nov 03 21:42:25.353178 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.353011    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zcbh\" (UniqueName: \"kubernetes.io/projected/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-kube-api-access-2zcbh\") pod \"maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9\" (UID: \"9bf836a1-d2b2-4f9b-bca1-92a40c71811b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"
Nov 03 21:42:25.667127 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.667095    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="720802a7-8dda-4dcc-8deb-d764518f49c1" path="/var/lib/kubelet/pods/720802a7-8dda-4dcc-8deb-d764518f49c1/volumes"
Nov 03 21:42:25.667682 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.667665    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74369a1d-5e9b-47a7-8a35-a6d2777e620b" path="/var/lib/kubelet/pods/74369a1d-5e9b-47a7-8a35-a6d2777e620b/volumes"
Nov 03 21:42:25.788043 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.788006    5053 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 03 21:42:25.788214 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.788064    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9" podUID="91eb34d0-5de4-4a33-b596-0f69a14d4131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 03 21:42:25.799928 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.799896    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-maas-api-tls\") pod \"maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9\" (UID: \"9bf836a1-d2b2-4f9b-bca1-92a40c71811b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"
Nov 03 21:42:25.825438 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:25.812948    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-maas-api-tls\") pod \"maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9\" (UID: \"9bf836a1-d2b2-4f9b-bca1-92a40c71811b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"
Nov 03 21:42:26.096654 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:26.096616    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"
Nov 03 21:42:26.506190 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:26.506155    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"]
Nov 03 21:42:27.176632 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:27.176597    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:27.177304 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:27.177190    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:27.179824 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:27.179785    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:42:27.259879 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:27.259842    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9" event={"ID":"9bf836a1-d2b2-4f9b-bca1-92a40c71811b","Type":"ContainerStarted","Data":"0e41bd9a98200b96723d03e4cf342af0bed1c693b231f300d3d28ef9943007a6"}
Nov 03 21:42:27.397309 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:27.397266    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"]
Nov 03 21:42:27.397499 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:27.397467    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9" podUID="7b508d7b-1822-47fa-9565-61da014df9c3" containerName="istio-proxy" containerID="cri-o://cb84fc8f0c298d92912fa7fa21ac1bd7ddf56b9d9115f2a684d8fce7c65c0d3d" gracePeriod=30
Nov 03 21:42:28.242875 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:28.242834    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j"]
Nov 03 21:42:28.248153 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:28.248126    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j"
Nov 03 21:42:28.299935 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:28.299902    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:42:28.301582 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:28.301552    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j"]
Nov 03 21:42:28.316703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:28.316669    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:42:28.365363 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:28.365329    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b576e48b-2608-4984-8630-4f449cdbe18a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j\" (UID: \"b576e48b-2608-4984-8630-4f449cdbe18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j"
Nov 03 21:42:28.365518 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:28.365424    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x948w\" (UniqueName: \"kubernetes.io/projected/b576e48b-2608-4984-8630-4f449cdbe18a-kube-api-access-x948w\") pod \"maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j\" (UID: \"b576e48b-2608-4984-8630-4f449cdbe18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j"
Nov 03 21:42:28.365671 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:28.365641    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b576e48b-2608-4984-8630-4f449cdbe18a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j\" (UID: \"b576e48b-2608-4984-8630-4f449cdbe18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j"
Nov 03 21:42:28.475289 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:28.472565    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b576e48b-2608-4984-8630-4f449cdbe18a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j\" (UID: \"b576e48b-2608-4984-8630-4f449cdbe18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j"
Nov 03 21:42:28.475289 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:28.472678    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x948w\" (UniqueName: \"kubernetes.io/projected/b576e48b-2608-4984-8630-4f449cdbe18a-kube-api-access-x948w\") pod \"maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j\" (UID: \"b576e48b-2608-4984-8630-4f449cdbe18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j"
Nov 03 21:42:28.475289 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:28.472757    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b576e48b-2608-4984-8630-4f449cdbe18a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j\" (UID: \"b576e48b-2608-4984-8630-4f449cdbe18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j"
Nov 03 21:42:28.483016 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:28.482984    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b576e48b-2608-4984-8630-4f449cdbe18a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j\" (UID: \"b576e48b-2608-4984-8630-4f449cdbe18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j"
Nov 03 21:42:28.492351 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:28.485859    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b576e48b-2608-4984-8630-4f449cdbe18a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j\" (UID: \"b576e48b-2608-4984-8630-4f449cdbe18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j"
Nov 03 21:42:28.529320 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:28.529223    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x948w\" (UniqueName: \"kubernetes.io/projected/b576e48b-2608-4984-8630-4f449cdbe18a-kube-api-access-x948w\") pod \"maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j\" (UID: \"b576e48b-2608-4984-8630-4f449cdbe18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j"
Nov 03 21:42:28.564745 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:28.564712    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j"
Nov 03 21:42:28.945231 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:28.945193    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j"]
Nov 03 21:42:28.946380 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:42:28.946354    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb576e48b_2608_4984_8630_4f449cdbe18a.slice/crio-c359b8a69647da06c4923bb27f60eaa24098d1b63721309dc1511a1f52d9e7b6 WatchSource:0}: Error finding container c359b8a69647da06c4923bb27f60eaa24098d1b63721309dc1511a1f52d9e7b6: Status 404 returned error can't find the container with id c359b8a69647da06c4923bb27f60eaa24098d1b63721309dc1511a1f52d9e7b6
Nov 03 21:42:29.276652 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:29.276621    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j" event={"ID":"b576e48b-2608-4984-8630-4f449cdbe18a","Type":"ContainerStarted","Data":"a76183f97b79d58f9c0dd4b91701b3a54f85fefdc99e5254c753a073b54377e0"}
Nov 03 21:42:29.277026 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:29.276676    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j" event={"ID":"b576e48b-2608-4984-8630-4f449cdbe18a","Type":"ContainerStarted","Data":"c359b8a69647da06c4923bb27f60eaa24098d1b63721309dc1511a1f52d9e7b6"}
Nov 03 21:42:29.389944 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:29.389891    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j" podStartSLOduration=1.389878264 podStartE2EDuration="1.389878264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:29.388337627 +0000 UTC m=+1663.882259503" watchObservedRunningTime="2026-11-03 21:42:29.389878264 +0000 UTC m=+1663.883800138"
Nov 03 21:42:29.799289 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:29.799251    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7682-336ee5"]
Nov 03 21:42:29.803614 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:29.803590    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7682-336ee5"
Nov 03 21:42:29.855833 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:29.855803    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8dgqp\""
Nov 03 21:42:29.856568 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:29.856540    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7682-336ee5"]
Nov 03 21:42:29.896029 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:29.895997    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwt42\" (UniqueName: \"kubernetes.io/projected/9e45f37f-b605-43c5-a1b9-87024c34cc33-kube-api-access-jwt42\") pod \"test-curl-7682-336ee5\" (UID: \"9e45f37f-b605-43c5-a1b9-87024c34cc33\") " pod="opendatahub/test-curl-7682-336ee5"
Nov 03 21:42:29.997586 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:29.997553    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwt42\" (UniqueName: \"kubernetes.io/projected/9e45f37f-b605-43c5-a1b9-87024c34cc33-kube-api-access-jwt42\") pod \"test-curl-7682-336ee5\" (UID: \"9e45f37f-b605-43c5-a1b9-87024c34cc33\") " pod="opendatahub/test-curl-7682-336ee5"
Nov 03 21:42:30.054306 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:30.054209    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwt42\" (UniqueName: \"kubernetes.io/projected/9e45f37f-b605-43c5-a1b9-87024c34cc33-kube-api-access-jwt42\") pod \"test-curl-7682-336ee5\" (UID: \"9e45f37f-b605-43c5-a1b9-87024c34cc33\") " pod="opendatahub/test-curl-7682-336ee5"
Nov 03 21:42:30.185359 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:30.185199    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7682-336ee5"
Nov 03 21:42:30.291373 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:30.291196    5053 generic.go:356] "Generic (PLEG): container finished" podID="b576e48b-2608-4984-8630-4f449cdbe18a" containerID="a76183f97b79d58f9c0dd4b91701b3a54f85fefdc99e5254c753a073b54377e0" exitCode=0
Nov 03 21:42:30.291373 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:30.291288    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j" event={"ID":"b576e48b-2608-4984-8630-4f449cdbe18a","Type":"ContainerDied","Data":"a76183f97b79d58f9c0dd4b91701b3a54f85fefdc99e5254c753a073b54377e0"}
Nov 03 21:42:30.568134 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:30.568092    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7682-336ee5"]
Nov 03 21:42:30.581742 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:42:30.581706    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e45f37f_b605_43c5_a1b9_87024c34cc33.slice/crio-006619b3a0506e69ff1516ae577882f6c1b7932d7603be631c72eb0fb16e01f1 WatchSource:0}: Error finding container 006619b3a0506e69ff1516ae577882f6c1b7932d7603be631c72eb0fb16e01f1: Status 404 returned error can't find the container with id 006619b3a0506e69ff1516ae577882f6c1b7932d7603be631c72eb0fb16e01f1
Nov 03 21:42:30.583910 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:30.583894    5053 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:42:31.301135 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:31.301093    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7682-336ee5" event={"ID":"9e45f37f-b605-43c5-a1b9-87024c34cc33","Type":"ContainerStarted","Data":"006619b3a0506e69ff1516ae577882f6c1b7932d7603be631c72eb0fb16e01f1"}
Nov 03 21:42:31.510710 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:31.510685    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j"
Nov 03 21:42:31.621885 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:31.621860    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b576e48b-2608-4984-8630-4f449cdbe18a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b576e48b-2608-4984-8630-4f449cdbe18a-maas-api-openshift-ca\") pod \"b576e48b-2608-4984-8630-4f449cdbe18a\" (UID: \"b576e48b-2608-4984-8630-4f449cdbe18a\") "
Nov 03 21:42:31.621963 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:31.621924    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b576e48b-2608-4984-8630-4f449cdbe18a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b576e48b-2608-4984-8630-4f449cdbe18a-maas-api-xks-ca\") pod \"b576e48b-2608-4984-8630-4f449cdbe18a\" (UID: \"b576e48b-2608-4984-8630-4f449cdbe18a\") "
Nov 03 21:42:31.622114 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:31.622097    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b576e48b-2608-4984-8630-4f449cdbe18a-kube-api-access-x948w\" (UniqueName: \"kubernetes.io/projected/b576e48b-2608-4984-8630-4f449cdbe18a-kube-api-access-x948w\") pod \"b576e48b-2608-4984-8630-4f449cdbe18a\" (UID: \"b576e48b-2608-4984-8630-4f449cdbe18a\") "
Nov 03 21:42:31.622305 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:31.622278    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b576e48b-2608-4984-8630-4f449cdbe18a-maas-api-openshift-ca" pod "b576e48b-2608-4984-8630-4f449cdbe18a" (UID: "b576e48b-2608-4984-8630-4f449cdbe18a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:42:31.622718 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:31.622694    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b576e48b-2608-4984-8630-4f449cdbe18a-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:31.624451 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:31.624415    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b576e48b-2608-4984-8630-4f449cdbe18a-maas-api-xks-ca" pod "b576e48b-2608-4984-8630-4f449cdbe18a" (UID: "b576e48b-2608-4984-8630-4f449cdbe18a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:42:31.624652 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:31.624480    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b576e48b-2608-4984-8630-4f449cdbe18a-kube-api-access-x948w" pod "b576e48b-2608-4984-8630-4f449cdbe18a" (UID: "b576e48b-2608-4984-8630-4f449cdbe18a"). InnerVolumeSpecName "kube-api-access-x948w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:31.724086 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:31.724048    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x948w\" (UniqueName: \"kubernetes.io/projected/b576e48b-2608-4984-8630-4f449cdbe18a-kube-api-access-x948w\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:31.724086 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:31.724082    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b576e48b-2608-4984-8630-4f449cdbe18a-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:32.314931 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.314887    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7682-336ee5" event={"ID":"9e45f37f-b605-43c5-a1b9-87024c34cc33","Type":"ContainerStarted","Data":"dd22e28dec6807fd93ef4302eddbfa61c30087e9076202d9827fe2441bf4786e"}
Nov 03 21:42:32.317283 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.317252    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j" event={"ID":"b576e48b-2608-4984-8630-4f449cdbe18a","Type":"ContainerDied","Data":"c359b8a69647da06c4923bb27f60eaa24098d1b63721309dc1511a1f52d9e7b6"}
Nov 03 21:42:32.317434 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.317316    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c359b8a69647da06c4923bb27f60eaa24098d1b63721309dc1511a1f52d9e7b6"
Nov 03 21:42:32.317434 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.317385    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j"
Nov 03 21:42:32.320723 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.320697    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9" event={"ID":"9bf836a1-d2b2-4f9b-bca1-92a40c71811b","Type":"ContainerStarted","Data":"7bcdf3debbd77a98cfbde34c3c3d3dfa5e942b7d02128408729d3ed93e832a7d"}
Nov 03 21:42:32.321111 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.321076    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"
Nov 03 21:42:32.428526 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.428479    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7682-336ee5" podStartSLOduration=2.119881414 podStartE2EDuration="3.428464701s" totalImagesPullingTime="1.308583287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:32.427561052 +0000 UTC m=+1666.921482928" watchObservedRunningTime="2026-11-03 21:42:32.428464701 +0000 UTC m=+1666.922386602"
Nov 03 21:42:32.542955 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.542896    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9" podStartSLOduration=2.435701696 podStartE2EDuration="7.542877797s" totalImagesPullingTime="5.107176101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:32.541265016 +0000 UTC m=+1667.035186892" watchObservedRunningTime="2026-11-03 21:42:32.542877797 +0000 UTC m=+1667.036799688"
Nov 03 21:42:32.691016 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.690991    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:42:32.854188 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.854102    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b508d7b-1822-47fa-9565-61da014df9c3-kube-api-access-xngk7\" (UniqueName: \"kubernetes.io/projected/7b508d7b-1822-47fa-9565-61da014df9c3-kube-api-access-xngk7\") pod \"7b508d7b-1822-47fa-9565-61da014df9c3\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") "
Nov 03 21:42:32.854188 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.854181    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b508d7b-1822-47fa-9565-61da014df9c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b508d7b-1822-47fa-9565-61da014df9c3-istio-podinfo\") pod \"7b508d7b-1822-47fa-9565-61da014df9c3\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") "
Nov 03 21:42:32.854464 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.854268    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-credential-socket\") pod \"7b508d7b-1822-47fa-9565-61da014df9c3\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") "
Nov 03 21:42:32.854464 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.854309    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b508d7b-1822-47fa-9565-61da014df9c3-istio-token\" (UniqueName: \"kubernetes.io/projected/7b508d7b-1822-47fa-9565-61da014df9c3-istio-token\") pod \"7b508d7b-1822-47fa-9565-61da014df9c3\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") "
Nov 03 21:42:32.854464 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.854362    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-istio-envoy\") pod \"7b508d7b-1822-47fa-9565-61da014df9c3\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") "
Nov 03 21:42:32.854624 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.854521    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-istio-data\") pod \"7b508d7b-1822-47fa-9565-61da014df9c3\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") "
Nov 03 21:42:32.854624 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.854554    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-credential-socket" pod "7b508d7b-1822-47fa-9565-61da014df9c3" (UID: "7b508d7b-1822-47fa-9565-61da014df9c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:32.854624 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.854572    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b508d7b-1822-47fa-9565-61da014df9c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b508d7b-1822-47fa-9565-61da014df9c3-istiod-ca-cert\") pod \"7b508d7b-1822-47fa-9565-61da014df9c3\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") "
Nov 03 21:42:32.854789 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.854637    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-workload-certs\") pod \"7b508d7b-1822-47fa-9565-61da014df9c3\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") "
Nov 03 21:42:32.854789 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.854724    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-workload-socket\") pod \"7b508d7b-1822-47fa-9565-61da014df9c3\" (UID: \"7b508d7b-1822-47fa-9565-61da014df9c3\") "
Nov 03 21:42:32.855066 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.854867    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-istio-data" pod "7b508d7b-1822-47fa-9565-61da014df9c3" (UID: "7b508d7b-1822-47fa-9565-61da014df9c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:32.855066 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.854928    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-workload-certs" pod "7b508d7b-1822-47fa-9565-61da014df9c3" (UID: "7b508d7b-1822-47fa-9565-61da014df9c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:32.855066 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.854984    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b508d7b-1822-47fa-9565-61da014df9c3-istiod-ca-cert" pod "7b508d7b-1822-47fa-9565-61da014df9c3" (UID: "7b508d7b-1822-47fa-9565-61da014df9c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:42:32.855323 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.855130    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-workload-socket" pod "7b508d7b-1822-47fa-9565-61da014df9c3" (UID: "7b508d7b-1822-47fa-9565-61da014df9c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:32.855732 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.855695    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:32.855732 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.855723    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b508d7b-1822-47fa-9565-61da014df9c3-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:32.855897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.855739    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:32.855897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.855752    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:32.855897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.855767    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:32.856976 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.856912    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b508d7b-1822-47fa-9565-61da014df9c3-istio-token" pod "7b508d7b-1822-47fa-9565-61da014df9c3" (UID: "7b508d7b-1822-47fa-9565-61da014df9c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:32.856976 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.856951    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b508d7b-1822-47fa-9565-61da014df9c3-kube-api-access-xngk7" pod "7b508d7b-1822-47fa-9565-61da014df9c3" (UID: "7b508d7b-1822-47fa-9565-61da014df9c3"). InnerVolumeSpecName "kube-api-access-xngk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:32.857103 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.856989    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b508d7b-1822-47fa-9565-61da014df9c3-istio-podinfo" pod "7b508d7b-1822-47fa-9565-61da014df9c3" (UID: "7b508d7b-1822-47fa-9565-61da014df9c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:42:32.857202 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.857185    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-istio-envoy" pod "7b508d7b-1822-47fa-9565-61da014df9c3" (UID: "7b508d7b-1822-47fa-9565-61da014df9c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:32.956880 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.956851    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xngk7\" (UniqueName: \"kubernetes.io/projected/7b508d7b-1822-47fa-9565-61da014df9c3-kube-api-access-xngk7\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:32.956880 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.956878    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b508d7b-1822-47fa-9565-61da014df9c3-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:32.956880 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.956887    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b508d7b-1822-47fa-9565-61da014df9c3-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:32.957111 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:32.956897    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b508d7b-1822-47fa-9565-61da014df9c3-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:33.341925 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:33.341886    5053 generic.go:356] "Generic (PLEG): container finished" podID="7b508d7b-1822-47fa-9565-61da014df9c3" containerID="cb84fc8f0c298d92912fa7fa21ac1bd7ddf56b9d9115f2a684d8fce7c65c0d3d" exitCode=0
Nov 03 21:42:33.342930 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:33.342894    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"
Nov 03 21:42:33.343065 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:33.343039    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9" event={"ID":"7b508d7b-1822-47fa-9565-61da014df9c3","Type":"ContainerDied","Data":"cb84fc8f0c298d92912fa7fa21ac1bd7ddf56b9d9115f2a684d8fce7c65c0d3d"}
Nov 03 21:42:33.343119 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:33.343086    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9" event={"ID":"7b508d7b-1822-47fa-9565-61da014df9c3","Type":"ContainerDied","Data":"f8d1fa4ec13856b98b2067c186ca6496bfb06d9faf9f8d3c9a9c7b1804b61b97"}
Nov 03 21:42:33.343119 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:33.343107    5053 scope.go:122] "RemoveContainer" containerID="cb84fc8f0c298d92912fa7fa21ac1bd7ddf56b9d9115f2a684d8fce7c65c0d3d"
Nov 03 21:42:33.357298 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:33.357268    5053 scope.go:122] "RemoveContainer" containerID="8b1315fa3b93f0a626aab55322e8fe8ac790bd1e711f4cbdfca21488ad0e7e3d"
Nov 03 21:42:33.367819 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:33.367787    5053 scope.go:122] "RemoveContainer" containerID="cb84fc8f0c298d92912fa7fa21ac1bd7ddf56b9d9115f2a684d8fce7c65c0d3d"
Nov 03 21:42:33.368092 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:42:33.368073    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb84fc8f0c298d92912fa7fa21ac1bd7ddf56b9d9115f2a684d8fce7c65c0d3d\": container with ID starting with cb84fc8f0c298d92912fa7fa21ac1bd7ddf56b9d9115f2a684d8fce7c65c0d3d not found: ID does not exist" containerID="cb84fc8f0c298d92912fa7fa21ac1bd7ddf56b9d9115f2a684d8fce7c65c0d3d"
Nov 03 21:42:33.368150 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:33.368099    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb84fc8f0c298d92912fa7fa21ac1bd7ddf56b9d9115f2a684d8fce7c65c0d3d"} err="failed to get container status \"cb84fc8f0c298d92912fa7fa21ac1bd7ddf56b9d9115f2a684d8fce7c65c0d3d\": rpc error: code = NotFound desc = could not find container \"cb84fc8f0c298d92912fa7fa21ac1bd7ddf56b9d9115f2a684d8fce7c65c0d3d\": container with ID starting with cb84fc8f0c298d92912fa7fa21ac1bd7ddf56b9d9115f2a684d8fce7c65c0d3d not found: ID does not exist"
Nov 03 21:42:33.368150 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:33.368117    5053 scope.go:122] "RemoveContainer" containerID="8b1315fa3b93f0a626aab55322e8fe8ac790bd1e711f4cbdfca21488ad0e7e3d"
Nov 03 21:42:33.368382 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:42:33.368366    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b1315fa3b93f0a626aab55322e8fe8ac790bd1e711f4cbdfca21488ad0e7e3d\": container with ID starting with 8b1315fa3b93f0a626aab55322e8fe8ac790bd1e711f4cbdfca21488ad0e7e3d not found: ID does not exist" containerID="8b1315fa3b93f0a626aab55322e8fe8ac790bd1e711f4cbdfca21488ad0e7e3d"
Nov 03 21:42:33.368447 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:33.368389    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b1315fa3b93f0a626aab55322e8fe8ac790bd1e711f4cbdfca21488ad0e7e3d"} err="failed to get container status \"8b1315fa3b93f0a626aab55322e8fe8ac790bd1e711f4cbdfca21488ad0e7e3d\": rpc error: code = NotFound desc = could not find container \"8b1315fa3b93f0a626aab55322e8fe8ac790bd1e711f4cbdfca21488ad0e7e3d\": container with ID starting with 8b1315fa3b93f0a626aab55322e8fe8ac790bd1e711f4cbdfca21488ad0e7e3d not found: ID does not exist"
Nov 03 21:42:33.512271 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:33.512214    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"]
Nov 03 21:42:33.564865 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:33.564815    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-29472d-openshift-default-55bd446969-f54r9"]
Nov 03 21:42:33.670956 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:33.670869    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b508d7b-1822-47fa-9565-61da014df9c3" path="/var/lib/kubelet/pods/7b508d7b-1822-47fa-9565-61da014df9c3/volumes"
Nov 03 21:42:33.990900 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:33.984614    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7682-336ee5"]
Nov 03 21:42:33.998342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:33.998309    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7682-336ee5"]
Nov 03 21:42:34.351632 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.351546    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7682-336ee5" podUID="9e45f37f-b605-43c5-a1b9-87024c34cc33" containerName="test-curl-7682-336ee5" containerID="cri-o://dd22e28dec6807fd93ef4302eddbfa61c30087e9076202d9827fe2441bf4786e" gracePeriod=2
Nov 03 21:42:34.393641 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.393608    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"]
Nov 03 21:42:34.394294 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.394278    5053 cpu_manager.go:414] "containerMap: removing container" podUID="7b508d7b-1822-47fa-9565-61da014df9c3" containerName="istio-proxy"
Nov 03 21:42:34.394294 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.394294    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b508d7b-1822-47fa-9565-61da014df9c3" containerName="istio-proxy"
Nov 03 21:42:34.394401 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.394320    5053 cpu_manager.go:414] "containerMap: removing container" podUID="9e45f37f-b605-43c5-a1b9-87024c34cc33" containerName="test-curl-7682-336ee5"
Nov 03 21:42:34.394401 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.394326    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e45f37f-b605-43c5-a1b9-87024c34cc33" containerName="test-curl-7682-336ee5"
Nov 03 21:42:34.394401 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.394336    5053 cpu_manager.go:414] "containerMap: removing container" podUID="b576e48b-2608-4984-8630-4f449cdbe18a" containerName="revoke-keys"
Nov 03 21:42:34.394401 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.394341    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b576e48b-2608-4984-8630-4f449cdbe18a" containerName="revoke-keys"
Nov 03 21:42:34.394529 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.394440    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b508d7b-1822-47fa-9565-61da014df9c3" containerName="istio-proxy"
Nov 03 21:42:34.394529 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.394451    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e45f37f-b605-43c5-a1b9-87024c34cc33" containerName="test-curl-7682-336ee5"
Nov 03 21:42:34.394529 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.394458    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="b576e48b-2608-4984-8630-4f449cdbe18a" containerName="revoke-keys"
Nov 03 21:42:34.399612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.399591    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.403281 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:42:34.403249    5053 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7682-336ee5\" is forbidden: User \"system:node:ip-10-0-128-43.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-43.us-west-1.compute.internal' and this object" podUID="9e45f37f-b605-43c5-a1b9-87024c34cc33" pod="opendatahub/test-curl-7682-336ee5"
Nov 03 21:42:34.485307 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.485269    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"]
Nov 03 21:42:34.544104 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.544071    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-shn5d"]
Nov 03 21:42:34.545076 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.545052    5053 cpu_manager.go:414] "containerMap: removing container" podUID="7b508d7b-1822-47fa-9565-61da014df9c3" containerName="istio-proxy"
Nov 03 21:42:34.545181 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.545079    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b508d7b-1822-47fa-9565-61da014df9c3" containerName="istio-proxy"
Nov 03 21:42:34.545301 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.545280    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b508d7b-1822-47fa-9565-61da014df9c3" containerName="istio-proxy"
Nov 03 21:42:34.557292 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.549597    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-shn5d"
Nov 03 21:42:34.569219 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:42:34.569186    5053 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7682-336ee5\" is forbidden: User \"system:node:ip-10-0-128-43.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-43.us-west-1.compute.internal' and this object" podUID="9e45f37f-b605-43c5-a1b9-87024c34cc33" pod="opendatahub/test-curl-7682-336ee5"
Nov 03 21:42:34.580478 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.580449    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.580588 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.580489    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2ngd\" (UniqueName: \"kubernetes.io/projected/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-kube-api-access-p2ngd\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.580631 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.580602    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.580674 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.580641    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.580743 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.580725    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.580865 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.580851    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.580910 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.580900    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.580945 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.580923    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.580945 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.580938    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.601465 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.601356    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-shn5d"]
Nov 03 21:42:34.601928 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.601869    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:42:34.602098 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.602083    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:42:34.682009 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.681977    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl2m7\" (UniqueName: \"kubernetes.io/projected/984b50dc-8d13-43f9-8cae-71bb34d2f96b-kube-api-access-pl2m7\") pod \"maas-api-revoke-keys-e2e-mt-c1662df5-shn5d\" (UID: \"984b50dc-8d13-43f9-8cae-71bb34d2f96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-shn5d"
Nov 03 21:42:34.682213 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.682036    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984b50dc-8d13-43f9-8cae-71bb34d2f96b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1662df5-shn5d\" (UID: \"984b50dc-8d13-43f9-8cae-71bb34d2f96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-shn5d"
Nov 03 21:42:34.682213 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.682161    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.682354 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.682230    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984b50dc-8d13-43f9-8cae-71bb34d2f96b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1662df5-shn5d\" (UID: \"984b50dc-8d13-43f9-8cae-71bb34d2f96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-shn5d"
Nov 03 21:42:34.682354 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.682272    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.682354 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.682323    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.682521 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.682352    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.682521 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.682406    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.682521 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.682430    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2ngd\" (UniqueName: \"kubernetes.io/projected/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-kube-api-access-p2ngd\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.682521 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.682516    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.682742 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.682542    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.682742 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.682562    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.682742 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.682676    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.682742 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.682675    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.683034 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.683008    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.683292 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.683209    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.683567 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.683315    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.684997 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.684978    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.685094 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.685074    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.730776 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:42:34.730733    5053 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7682-336ee5\" is forbidden: User \"system:node:ip-10-0-128-43.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-43.us-west-1.compute.internal' and this object" podUID="9e45f37f-b605-43c5-a1b9-87024c34cc33" pod="opendatahub/test-curl-7682-336ee5"
Nov 03 21:42:34.739022 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.738993    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.784577 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.784545    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984b50dc-8d13-43f9-8cae-71bb34d2f96b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1662df5-shn5d\" (UID: \"984b50dc-8d13-43f9-8cae-71bb34d2f96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-shn5d"
Nov 03 21:42:34.784740 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.784647    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984b50dc-8d13-43f9-8cae-71bb34d2f96b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1662df5-shn5d\" (UID: \"984b50dc-8d13-43f9-8cae-71bb34d2f96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-shn5d"
Nov 03 21:42:34.784980 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.784955    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl2m7\" (UniqueName: \"kubernetes.io/projected/984b50dc-8d13-43f9-8cae-71bb34d2f96b-kube-api-access-pl2m7\") pod \"maas-api-revoke-keys-e2e-mt-c1662df5-shn5d\" (UID: \"984b50dc-8d13-43f9-8cae-71bb34d2f96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-shn5d"
Nov 03 21:42:34.785668 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.785634    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984b50dc-8d13-43f9-8cae-71bb34d2f96b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1662df5-shn5d\" (UID: \"984b50dc-8d13-43f9-8cae-71bb34d2f96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-shn5d"
Nov 03 21:42:34.787291 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.787270    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984b50dc-8d13-43f9-8cae-71bb34d2f96b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1662df5-shn5d\" (UID: \"984b50dc-8d13-43f9-8cae-71bb34d2f96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-shn5d"
Nov 03 21:42:34.787630 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.787590    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2ngd\" (UniqueName: \"kubernetes.io/projected/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-kube-api-access-p2ngd\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:34.842030 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.842001    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl2m7\" (UniqueName: \"kubernetes.io/projected/984b50dc-8d13-43f9-8cae-71bb34d2f96b-kube-api-access-pl2m7\") pod \"maas-api-revoke-keys-e2e-mt-c1662df5-shn5d\" (UID: \"984b50dc-8d13-43f9-8cae-71bb34d2f96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-shn5d"
Nov 03 21:42:34.865801 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:34.865739    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-shn5d"
Nov 03 21:42:35.017670 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:35.017616    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:35.246668 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:35.246640    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-shn5d"]
Nov 03 21:42:35.247909 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:42:35.247885    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984b50dc_8d13_43f9_8cae_71bb34d2f96b.slice/crio-8bebd09cc14c7dd03e76d4bb0e84c8016792ab10322720bd5a0075dbd870efc5 WatchSource:0}: Error finding container 8bebd09cc14c7dd03e76d4bb0e84c8016792ab10322720bd5a0075dbd870efc5: Status 404 returned error can't find the container with id 8bebd09cc14c7dd03e76d4bb0e84c8016792ab10322720bd5a0075dbd870efc5
Nov 03 21:42:35.361458 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:35.361420    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-shn5d" event={"ID":"984b50dc-8d13-43f9-8cae-71bb34d2f96b","Type":"ContainerStarted","Data":"8bebd09cc14c7dd03e76d4bb0e84c8016792ab10322720bd5a0075dbd870efc5"}
Nov 03 21:42:35.401407 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:35.401377    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"]
Nov 03 21:42:35.402394 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:42:35.402367    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdeb5c9c_5820_476c_9763_5ccad90f0c5d.slice/crio-7e37470aa296acf9c962e4c0b430bd63c28d38e64e2e27acb42bd294bcf67bae WatchSource:0}: Error finding container 7e37470aa296acf9c962e4c0b430bd63c28d38e64e2e27acb42bd294bcf67bae: Status 404 returned error can't find the container with id 7e37470aa296acf9c962e4c0b430bd63c28d38e64e2e27acb42bd294bcf67bae
Nov 03 21:42:35.404483 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:35.404451    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:42:35.404610 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:35.404518    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:42:35.404610 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:35.404546    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:42:35.733642 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:42:35.733588    5053 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7682-336ee5\" is forbidden: User \"system:node:ip-10-0-128-43.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-43.us-west-1.compute.internal' and this object" podUID="9e45f37f-b605-43c5-a1b9-87024c34cc33" pod="opendatahub/test-curl-7682-336ee5"
Nov 03 21:42:36.376554 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:36.375430    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2" event={"ID":"bdeb5c9c-5820-476c-9763-5ccad90f0c5d","Type":"ContainerStarted","Data":"41078f5752bd77d5f41000cca4db67ea60a0da238e3c87936e41a9dcb8019c8f"}
Nov 03 21:42:36.376554 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:36.375499    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2" event={"ID":"bdeb5c9c-5820-476c-9763-5ccad90f0c5d","Type":"ContainerStarted","Data":"7e37470aa296acf9c962e4c0b430bd63c28d38e64e2e27acb42bd294bcf67bae"}
Nov 03 21:42:36.382982 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:36.382581    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-shn5d" event={"ID":"984b50dc-8d13-43f9-8cae-71bb34d2f96b","Type":"ContainerStarted","Data":"dc42239d89451d040ec7b83e7c250232e94d3cad85eee8ad3a86b75199eef926"}
Nov 03 21:42:36.487195 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:42:36.487143    5053 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984b50dc_8d13_43f9_8cae_71bb34d2f96b.slice/crio-conmon-dc42239d89451d040ec7b83e7c250232e94d3cad85eee8ad3a86b75199eef926.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e45f37f_b605_43c5_a1b9_87024c34cc33.slice/crio-dd22e28dec6807fd93ef4302eddbfa61c30087e9076202d9827fe2441bf4786e.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 21:42:36.530485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:36.530422    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2" podStartSLOduration=2.530401367 podStartE2EDuration="2.530401367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:36.496875365 +0000 UTC m=+1670.990797241" watchObservedRunningTime="2026-11-03 21:42:36.530401367 +0000 UTC m=+1671.024323243"
Nov 03 21:42:36.698253 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:36.698213    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7682-336ee5"
Nov 03 21:42:36.858428 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:36.858347    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e45f37f-b605-43c5-a1b9-87024c34cc33-kube-api-access-jwt42\" (UniqueName: \"kubernetes.io/projected/9e45f37f-b605-43c5-a1b9-87024c34cc33-kube-api-access-jwt42\") pod \"9e45f37f-b605-43c5-a1b9-87024c34cc33\" (UID: \"9e45f37f-b605-43c5-a1b9-87024c34cc33\") "
Nov 03 21:42:36.860594 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:36.860570    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e45f37f-b605-43c5-a1b9-87024c34cc33-kube-api-access-jwt42" pod "9e45f37f-b605-43c5-a1b9-87024c34cc33" (UID: "9e45f37f-b605-43c5-a1b9-87024c34cc33"). InnerVolumeSpecName "kube-api-access-jwt42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:36.959740 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:36.959704    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwt42\" (UniqueName: \"kubernetes.io/projected/9e45f37f-b605-43c5-a1b9-87024c34cc33-kube-api-access-jwt42\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:37.023260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:37.022304    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:37.030259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:37.023657    5053 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 03 21:42:37.030259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:37.023713    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2" podUID="bdeb5c9c-5820-476c-9763-5ccad90f0c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 03 21:42:37.391426 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:37.391392    5053 generic.go:356] "Generic (PLEG): container finished" podID="9e45f37f-b605-43c5-a1b9-87024c34cc33" containerID="dd22e28dec6807fd93ef4302eddbfa61c30087e9076202d9827fe2441bf4786e" exitCode=137
Nov 03 21:42:37.391911 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:37.391441    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7682-336ee5"
Nov 03 21:42:37.391911 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:37.391492    5053 scope.go:122] "RemoveContainer" containerID="dd22e28dec6807fd93ef4302eddbfa61c30087e9076202d9827fe2441bf4786e"
Nov 03 21:42:37.393281 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:37.393254    5053 generic.go:356] "Generic (PLEG): container finished" podID="984b50dc-8d13-43f9-8cae-71bb34d2f96b" containerID="dc42239d89451d040ec7b83e7c250232e94d3cad85eee8ad3a86b75199eef926" exitCode=0
Nov 03 21:42:37.393422 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:37.393299    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-shn5d" event={"ID":"984b50dc-8d13-43f9-8cae-71bb34d2f96b","Type":"ContainerDied","Data":"dc42239d89451d040ec7b83e7c250232e94d3cad85eee8ad3a86b75199eef926"}
Nov 03 21:42:37.440261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:37.439408    5053 scope.go:122] "RemoveContainer" containerID="dd22e28dec6807fd93ef4302eddbfa61c30087e9076202d9827fe2441bf4786e"
Nov 03 21:42:37.440261 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:42:37.439870    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd22e28dec6807fd93ef4302eddbfa61c30087e9076202d9827fe2441bf4786e\": container with ID starting with dd22e28dec6807fd93ef4302eddbfa61c30087e9076202d9827fe2441bf4786e not found: ID does not exist" containerID="dd22e28dec6807fd93ef4302eddbfa61c30087e9076202d9827fe2441bf4786e"
Nov 03 21:42:37.440261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:37.439906    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd22e28dec6807fd93ef4302eddbfa61c30087e9076202d9827fe2441bf4786e"} err="failed to get container status \"dd22e28dec6807fd93ef4302eddbfa61c30087e9076202d9827fe2441bf4786e\": rpc error: code = NotFound desc = could not find container \"dd22e28dec6807fd93ef4302eddbfa61c30087e9076202d9827fe2441bf4786e\": container with ID starting with dd22e28dec6807fd93ef4302eddbfa61c30087e9076202d9827fe2441bf4786e not found: ID does not exist"
Nov 03 21:42:37.451574 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:42:37.451529    5053 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7682-336ee5\" is forbidden: User \"system:node:ip-10-0-128-43.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-43.us-west-1.compute.internal' and this object" podUID="9e45f37f-b605-43c5-a1b9-87024c34cc33" pod="opendatahub/test-curl-7682-336ee5"
Nov 03 21:42:37.673074 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:37.673042    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e45f37f-b605-43c5-a1b9-87024c34cc33" path="/var/lib/kubelet/pods/9e45f37f-b605-43c5-a1b9-87024c34cc33/volumes"
Nov 03 21:42:37.722272 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:37.719736    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-shn5d"
Nov 03 21:42:37.736828 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:37.736796    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f"]
Nov 03 21:42:37.753222 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:37.753178    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5025c-lhf4f"]
Nov 03 21:42:37.869706 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:37.869245    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/984b50dc-8d13-43f9-8cae-71bb34d2f96b-kube-api-access-pl2m7\" (UniqueName: \"kubernetes.io/projected/984b50dc-8d13-43f9-8cae-71bb34d2f96b-kube-api-access-pl2m7\") pod \"984b50dc-8d13-43f9-8cae-71bb34d2f96b\" (UID: \"984b50dc-8d13-43f9-8cae-71bb34d2f96b\") "
Nov 03 21:42:37.869706 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:37.869575    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/984b50dc-8d13-43f9-8cae-71bb34d2f96b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984b50dc-8d13-43f9-8cae-71bb34d2f96b-maas-api-xks-ca\") pod \"984b50dc-8d13-43f9-8cae-71bb34d2f96b\" (UID: \"984b50dc-8d13-43f9-8cae-71bb34d2f96b\") "
Nov 03 21:42:37.869706 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:37.869661    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/984b50dc-8d13-43f9-8cae-71bb34d2f96b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984b50dc-8d13-43f9-8cae-71bb34d2f96b-maas-api-openshift-ca\") pod \"984b50dc-8d13-43f9-8cae-71bb34d2f96b\" (UID: \"984b50dc-8d13-43f9-8cae-71bb34d2f96b\") "
Nov 03 21:42:37.870293 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:37.870224    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/984b50dc-8d13-43f9-8cae-71bb34d2f96b-maas-api-openshift-ca" pod "984b50dc-8d13-43f9-8cae-71bb34d2f96b" (UID: "984b50dc-8d13-43f9-8cae-71bb34d2f96b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:42:37.872602 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:37.872573    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/984b50dc-8d13-43f9-8cae-71bb34d2f96b-maas-api-xks-ca" pod "984b50dc-8d13-43f9-8cae-71bb34d2f96b" (UID: "984b50dc-8d13-43f9-8cae-71bb34d2f96b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:42:37.875829 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:37.875797    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984b50dc-8d13-43f9-8cae-71bb34d2f96b-kube-api-access-pl2m7" pod "984b50dc-8d13-43f9-8cae-71bb34d2f96b" (UID: "984b50dc-8d13-43f9-8cae-71bb34d2f96b"). InnerVolumeSpecName "kube-api-access-pl2m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:37.972421 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:37.972387    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984b50dc-8d13-43f9-8cae-71bb34d2f96b-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:37.972421 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:37.972422    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl2m7\" (UniqueName: \"kubernetes.io/projected/984b50dc-8d13-43f9-8cae-71bb34d2f96b-kube-api-access-pl2m7\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:37.972630 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:37.972439    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984b50dc-8d13-43f9-8cae-71bb34d2f96b-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:38.031263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:38.030678    5053 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 03 21:42:38.031263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:38.030741    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2" podUID="bdeb5c9c-5820-476c-9763-5ccad90f0c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 03 21:42:38.350132 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:38.350056    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"
Nov 03 21:42:38.423262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:38.421773    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-shn5d"
Nov 03 21:42:38.436262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:38.431346    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-shn5d" event={"ID":"984b50dc-8d13-43f9-8cae-71bb34d2f96b","Type":"ContainerDied","Data":"8bebd09cc14c7dd03e76d4bb0e84c8016792ab10322720bd5a0075dbd870efc5"}
Nov 03 21:42:38.436262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:38.431414    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bebd09cc14c7dd03e76d4bb0e84c8016792ab10322720bd5a0075dbd870efc5"
Nov 03 21:42:39.021105 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:39.020651    5053 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 03 21:42:39.021105 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:39.020725    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2" podUID="bdeb5c9c-5820-476c-9763-5ccad90f0c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 03 21:42:39.673493 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:39.673453    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d8a1267-4c43-4f54-815d-79e78fef26a5" path="/var/lib/kubelet/pods/2d8a1267-4c43-4f54-815d-79e78fef26a5/volumes"
Nov 03 21:42:40.026706 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:40.026669    5053 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 03 21:42:40.026938 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:40.026734    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2" podUID="bdeb5c9c-5820-476c-9763-5ccad90f0c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 03 21:42:40.959002 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:42:40.958972    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:42:41.018690 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:41.018651    5053 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 03 21:42:41.018969 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:41.018938    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2" podUID="bdeb5c9c-5820-476c-9763-5ccad90f0c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 03 21:42:42.203100 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:42.203068    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:42.203745 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:42.203717    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:42.205893 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:42.205865    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:42:45.237998 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:45.237949    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad161-m2jq8"]
Nov 03 21:42:45.239211 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:45.239190    5053 cpu_manager.go:414] "containerMap: removing container" podUID="984b50dc-8d13-43f9-8cae-71bb34d2f96b" containerName="revoke-keys"
Nov 03 21:42:45.239304 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:45.239214    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="984b50dc-8d13-43f9-8cae-71bb34d2f96b" containerName="revoke-keys"
Nov 03 21:42:45.239434 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:45.239420    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="984b50dc-8d13-43f9-8cae-71bb34d2f96b" containerName="revoke-keys"
Nov 03 21:42:45.244445 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:45.244425    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad161-m2jq8"
Nov 03 21:42:45.297228 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:45.297189    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad161-m2jq8"]
Nov 03 21:42:45.342210 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:45.342183    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:42:45.342705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:45.342689    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:42:45.353230 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:45.353198    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5nk6\" (UniqueName: \"kubernetes.io/projected/0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7-kube-api-access-g5nk6\") pod \"maas-api-revoke-keys-e2e-api-b-dad161-m2jq8\" (UID: \"0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad161-m2jq8"
Nov 03 21:42:45.353416 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:45.353260    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dad161-m2jq8\" (UID: \"0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad161-m2jq8"
Nov 03 21:42:45.353416 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:45.353317    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dad161-m2jq8\" (UID: \"0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad161-m2jq8"
Nov 03 21:42:45.454562 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:45.454520    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5nk6\" (UniqueName: \"kubernetes.io/projected/0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7-kube-api-access-g5nk6\") pod \"maas-api-revoke-keys-e2e-api-b-dad161-m2jq8\" (UID: \"0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad161-m2jq8"
Nov 03 21:42:45.454765 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:45.454579    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dad161-m2jq8\" (UID: \"0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad161-m2jq8"
Nov 03 21:42:45.454765 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:45.454645    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dad161-m2jq8\" (UID: \"0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad161-m2jq8"
Nov 03 21:42:45.455333 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:45.455313    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dad161-m2jq8\" (UID: \"0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad161-m2jq8"
Nov 03 21:42:45.457063 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:45.457040    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dad161-m2jq8\" (UID: \"0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad161-m2jq8"
Nov 03 21:42:45.527066 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:45.526976    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5nk6\" (UniqueName: \"kubernetes.io/projected/0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7-kube-api-access-g5nk6\") pod \"maas-api-revoke-keys-e2e-api-b-dad161-m2jq8\" (UID: \"0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad161-m2jq8"
Nov 03 21:42:45.577403 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:45.577374    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad161-m2jq8"
Nov 03 21:42:45.982873 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:45.982847    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad161-m2jq8"]
Nov 03 21:42:45.983852 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:42:45.983826    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2d21eb_e40d_4d48_a8b8_39bf88f1ddf7.slice/crio-61f66d3a57d6a107610aa1b8942566477955905bdfc68ec7233283bdfbb5de32 WatchSource:0}: Error finding container 61f66d3a57d6a107610aa1b8942566477955905bdfc68ec7233283bdfbb5de32: Status 404 returned error can't find the container with id 61f66d3a57d6a107610aa1b8942566477955905bdfc68ec7233283bdfbb5de32
Nov 03 21:42:46.540323 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:46.540211    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad161-m2jq8" event={"ID":"0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7","Type":"ContainerStarted","Data":"61f66d3a57d6a107610aa1b8942566477955905bdfc68ec7233283bdfbb5de32"}
Nov 03 21:42:46.840640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:46.840553    5053 scope.go:122] "RemoveContainer" containerID="81f83ba8c529028aec9e4bf2f85aa5df43924bd8edd9d6b69c8af737e74b5cc3"
Nov 03 21:42:46.852614 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:46.852590    5053 scope.go:122] "RemoveContainer" containerID="4518044edfbf9a3d2b488bd86268fccbaaccd9881f7d17121dfa9752e9580eeb"
Nov 03 21:42:47.550328 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:47.550212    5053 generic.go:356] "Generic (PLEG): container finished" podID="0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7" containerID="e7df406340219f4287ba7dd8ef0fad0237c20f38052f0bfad894ddf7dd3e886a" exitCode=0
Nov 03 21:42:47.550328 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:47.550298    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad161-m2jq8" event={"ID":"0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7","Type":"ContainerDied","Data":"e7df406340219f4287ba7dd8ef0fad0237c20f38052f0bfad894ddf7dd3e886a"}
Nov 03 21:42:48.749689 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:48.749655    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad161-m2jq8"
Nov 03 21:42:48.911317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:48.908407    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7-maas-api-xks-ca\") pod \"0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7\" (UID: \"0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7\") "
Nov 03 21:42:48.911317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:48.908531    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7-maas-api-openshift-ca\") pod \"0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7\" (UID: \"0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7\") "
Nov 03 21:42:48.911317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:48.908624    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7-kube-api-access-g5nk6\" (UniqueName: \"kubernetes.io/projected/0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7-kube-api-access-g5nk6\") pod \"0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7\" (UID: \"0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7\") "
Nov 03 21:42:48.946925 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:48.946887    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7-maas-api-openshift-ca" pod "0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7" (UID: "0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:42:48.963272 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:48.963219    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7-maas-api-xks-ca" pod "0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7" (UID: "0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:42:48.963614 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:48.963573    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7-kube-api-access-g5nk6" pod "0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7" (UID: "0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7"). InnerVolumeSpecName "kube-api-access-g5nk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:49.010737 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.010645    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:49.010737 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.010685    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5nk6\" (UniqueName: \"kubernetes.io/projected/0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7-kube-api-access-g5nk6\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:49.010737 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.010701    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:49.398970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.398933    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"]
Nov 03 21:42:49.399673 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.399652    5053 cpu_manager.go:414] "containerMap: removing container" podUID="0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7" containerName="revoke-keys"
Nov 03 21:42:49.399813 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.399676    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7" containerName="revoke-keys"
Nov 03 21:42:49.399913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.399898    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7" containerName="revoke-keys"
Nov 03 21:42:49.425831 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.425798    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.518685 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.518640    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.518879 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.518730    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.518879 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.518778    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmr6n\" (UniqueName: \"kubernetes.io/projected/a3694a1a-4c11-45b5-aed6-2568770e962c-kube-api-access-hmr6n\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.518879 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.518812    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.518879 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.518842    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.519059 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.518886    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.519059 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.518946    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3694a1a-4c11-45b5-aed6-2568770e962c-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.519059 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.519000    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.519187 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.519080    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.537312 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.533874    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"]
Nov 03 21:42:49.569215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.569183    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad161-m2jq8"
Nov 03 21:42:49.569421 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.569217    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad161-m2jq8" event={"ID":"0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7","Type":"ContainerDied","Data":"61f66d3a57d6a107610aa1b8942566477955905bdfc68ec7233283bdfbb5de32"}
Nov 03 21:42:49.569421 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.569283    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61f66d3a57d6a107610aa1b8942566477955905bdfc68ec7233283bdfbb5de32"
Nov 03 21:42:49.619998 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.619943    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3694a1a-4c11-45b5-aed6-2568770e962c-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.619998 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.620000    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.620411 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.620055    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.620411 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.620128    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.620411 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.620189    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.620411 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.620249    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmr6n\" (UniqueName: \"kubernetes.io/projected/a3694a1a-4c11-45b5-aed6-2568770e962c-kube-api-access-hmr6n\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.620411 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.620286    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.620411 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.620305    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.620411 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.620342    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.620901 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.620683    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.620901 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.620699    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.621008 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.620900    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3694a1a-4c11-45b5-aed6-2568770e962c-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.621008 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.620912    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.621140 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.621111    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.631501 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.631469    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.631681 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.631588    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.677211 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.677139    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmr6n\" (UniqueName: \"kubernetes.io/projected/a3694a1a-4c11-45b5-aed6-2568770e962c-kube-api-access-hmr6n\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.677211 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.677168    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:49.750081 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:49.750032    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:50.190286 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:42:50.190216    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3694a1a_4c11_45b5_aed6_2568770e962c.slice/crio-35f43826d7946f7b1a43d389cf82f162b7c02a88e0abe4c58fa1ddea839055bd WatchSource:0}: Error finding container 35f43826d7946f7b1a43d389cf82f162b7c02a88e0abe4c58fa1ddea839055bd: Status 404 returned error can't find the container with id 35f43826d7946f7b1a43d389cf82f162b7c02a88e0abe4c58fa1ddea839055bd
Nov 03 21:42:50.191523 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:50.191494    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"]
Nov 03 21:42:50.193346 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:50.193308    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:42:50.193446 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:50.193390    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:42:50.193446 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:50.193433    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:42:50.581172 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:50.581138    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72" event={"ID":"a3694a1a-4c11-45b5-aed6-2568770e962c","Type":"ContainerStarted","Data":"466c43ebd59361ffdfd74d1b6f6f33c5d6cb7087ec1535d21cef1254be173dd5"}
Nov 03 21:42:50.581398 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:50.581197    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72" event={"ID":"a3694a1a-4c11-45b5-aed6-2568770e962c","Type":"ContainerStarted","Data":"35f43826d7946f7b1a43d389cf82f162b7c02a88e0abe4c58fa1ddea839055bd"}
Nov 03 21:42:50.698015 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:50.697959    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72" podStartSLOduration=1.697941826 podStartE2EDuration="1.697941826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:50.695008599 +0000 UTC m=+1685.188930473" watchObservedRunningTime="2026-11-03 21:42:50.697941826 +0000 UTC m=+1685.191863754"
Nov 03 21:42:50.750965 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:50.750934    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:51.752936 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:51.752893    5053 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 03 21:42:51.753401 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:51.752956    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72" podUID="a3694a1a-4c11-45b5-aed6-2568770e962c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 03 21:42:52.537575 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:52.537405    5053 patch_prober.go:29] interesting pod/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:42:52.537575 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:52.537490    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt" podUID="46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:42:52.753986 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:52.753682    5053 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 03 21:42:52.753986 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:52.753765    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72" podUID="a3694a1a-4c11-45b5-aed6-2568770e962c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 03 21:42:53.628366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:53.628329    5053 generic.go:356] "Generic (PLEG): container finished" podID="46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" containerID="ea2b8ae0e04cdd2189d3750064ed76bebcdc91dac800f1fa3721b6d91000fe17" exitCode=0
Nov 03 21:42:53.628576 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:53.628440    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt" event={"ID":"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c","Type":"ContainerDied","Data":"ea2b8ae0e04cdd2189d3750064ed76bebcdc91dac800f1fa3721b6d91000fe17"}
Nov 03 21:42:53.628858 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:53.628842    5053 scope.go:122] "RemoveContainer" containerID="ea2b8ae0e04cdd2189d3750064ed76bebcdc91dac800f1fa3721b6d91000fe17"
Nov 03 21:42:53.630455 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:53.630425    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:42:53.630520 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:53.630500    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:42:53.630572 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:53.630532    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:42:53.752206 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:53.752147    5053 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 03 21:42:53.753011 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:53.752973    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72" podUID="a3694a1a-4c11-45b5-aed6-2568770e962c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 03 21:42:54.664866 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:54.664825    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt" event={"ID":"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c","Type":"ContainerStarted","Data":"8120e55e4dbee02ec044a208e8739b9815107a0696f315f354ca5f5cf073cad3"}
Nov 03 21:42:54.753263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:54.752867    5053 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 03 21:42:54.753263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:54.752952    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72" podUID="a3694a1a-4c11-45b5-aed6-2568770e962c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 03 21:42:55.518262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:55.509676    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:42:55.518262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:55.509805    5053 patch_prober.go:29] interesting pod/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 03 21:42:55.518262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:55.509846    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt" podUID="46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 03 21:42:55.754044 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:55.754005    5053 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 03 21:42:55.754687 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:55.754653    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72" podUID="a3694a1a-4c11-45b5-aed6-2568770e962c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 03 21:42:56.516603 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:56.516568    5053 patch_prober.go:29] interesting pod/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 03 21:42:56.516895 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:56.516864    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt" podUID="46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 03 21:42:56.893681 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:56.893596    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:56.894138 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:56.894076    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:56.894815 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:56.894795    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:42:57.525677 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:57.525644    5053 patch_prober.go:29] interesting pod/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 03 21:42:57.525979 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:57.525947    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt" podUID="46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 03 21:42:58.510675 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:58.510642    5053 patch_prober.go:29] interesting pod/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 03 21:42:58.511282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:58.511228    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt" podUID="46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 03 21:42:59.508693 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:59.508652    5053 patch_prober.go:29] interesting pod/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 03 21:42:59.508978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:42:59.508949    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt" podUID="46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 03 21:43:00.512393 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:00.512354    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:43:00.512909 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:00.512738    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:43:00.513440 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:00.513421    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:43:10.929580 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:43:10.929534    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:43:24.208608 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:24.208574    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-shn5d"]
Nov 03 21:43:24.257881 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:24.257847    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1662df5-shn5d"]
Nov 03 21:43:25.668151 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:25.668118    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="984b50dc-8d13-43f9-8cae-71bb34d2f96b" path="/var/lib/kubelet/pods/984b50dc-8d13-43f9-8cae-71bb34d2f96b/volumes"
Nov 03 21:43:31.621297 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:31.621257    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fff189-qsdv2"]
Nov 03 21:43:31.628054 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:31.628028    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fff189-qsdv2"
Nov 03 21:43:31.679479 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:31.679443    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fff189-qsdv2"]
Nov 03 21:43:31.701607 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:31.701568    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqwf8\" (UniqueName: \"kubernetes.io/projected/da49a5be-afe7-47a7-a0a5-5f05ca7d9231-kube-api-access-qqwf8\") pod \"maas-api-revoke-keys-e2e-api-a-fff189-qsdv2\" (UID: \"da49a5be-afe7-47a7-a0a5-5f05ca7d9231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fff189-qsdv2"
Nov 03 21:43:31.701809 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:31.701704    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da49a5be-afe7-47a7-a0a5-5f05ca7d9231-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fff189-qsdv2\" (UID: \"da49a5be-afe7-47a7-a0a5-5f05ca7d9231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fff189-qsdv2"
Nov 03 21:43:31.701923 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:31.701902    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da49a5be-afe7-47a7-a0a5-5f05ca7d9231-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fff189-qsdv2\" (UID: \"da49a5be-afe7-47a7-a0a5-5f05ca7d9231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fff189-qsdv2"
Nov 03 21:43:31.723695 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:31.723667    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:43:31.723880 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:31.723811    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:43:31.803528 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:31.803493    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da49a5be-afe7-47a7-a0a5-5f05ca7d9231-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fff189-qsdv2\" (UID: \"da49a5be-afe7-47a7-a0a5-5f05ca7d9231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fff189-qsdv2"
Nov 03 21:43:31.803703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:31.803612    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqwf8\" (UniqueName: \"kubernetes.io/projected/da49a5be-afe7-47a7-a0a5-5f05ca7d9231-kube-api-access-qqwf8\") pod \"maas-api-revoke-keys-e2e-api-a-fff189-qsdv2\" (UID: \"da49a5be-afe7-47a7-a0a5-5f05ca7d9231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fff189-qsdv2"
Nov 03 21:43:31.803703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:31.803646    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da49a5be-afe7-47a7-a0a5-5f05ca7d9231-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fff189-qsdv2\" (UID: \"da49a5be-afe7-47a7-a0a5-5f05ca7d9231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fff189-qsdv2"
Nov 03 21:43:31.804347 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:31.804319    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da49a5be-afe7-47a7-a0a5-5f05ca7d9231-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fff189-qsdv2\" (UID: \"da49a5be-afe7-47a7-a0a5-5f05ca7d9231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fff189-qsdv2"
Nov 03 21:43:31.806085 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:31.806066    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da49a5be-afe7-47a7-a0a5-5f05ca7d9231-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fff189-qsdv2\" (UID: \"da49a5be-afe7-47a7-a0a5-5f05ca7d9231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fff189-qsdv2"
Nov 03 21:43:31.859896 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:31.859866    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqwf8\" (UniqueName: \"kubernetes.io/projected/da49a5be-afe7-47a7-a0a5-5f05ca7d9231-kube-api-access-qqwf8\") pod \"maas-api-revoke-keys-e2e-api-a-fff189-qsdv2\" (UID: \"da49a5be-afe7-47a7-a0a5-5f05ca7d9231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fff189-qsdv2"
Nov 03 21:43:31.944616 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:31.944588    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fff189-qsdv2"
Nov 03 21:43:32.324727 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:32.324693    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fff189-qsdv2"]
Nov 03 21:43:32.326627 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:43:32.326590    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda49a5be_afe7_47a7_a0a5_5f05ca7d9231.slice/crio-eff3e5f24f1f17025e5e3821ab3708bc04808b6855b8fedb887f98353323ae76 WatchSource:0}: Error finding container eff3e5f24f1f17025e5e3821ab3708bc04808b6855b8fedb887f98353323ae76: Status 404 returned error can't find the container with id eff3e5f24f1f17025e5e3821ab3708bc04808b6855b8fedb887f98353323ae76
Nov 03 21:43:32.967653 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:32.967619    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fff189-qsdv2" event={"ID":"da49a5be-afe7-47a7-a0a5-5f05ca7d9231","Type":"ContainerStarted","Data":"b639292af1a02a12d4f09ca3a8408cdc737a73c6ae0621c068b2cf140cfee4e6"}
Nov 03 21:43:32.968075 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:32.967677    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fff189-qsdv2" event={"ID":"da49a5be-afe7-47a7-a0a5-5f05ca7d9231","Type":"ContainerStarted","Data":"eff3e5f24f1f17025e5e3821ab3708bc04808b6855b8fedb887f98353323ae76"}
Nov 03 21:43:33.090676 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:33.090616    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fff189-qsdv2" podStartSLOduration=2.090595331 podStartE2EDuration="2.090595331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:43:33.083714165 +0000 UTC m=+1727.577636041" watchObservedRunningTime="2026-11-03 21:43:33.090595331 +0000 UTC m=+1727.584517207"
Nov 03 21:43:33.975927 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:33.975884    5053 generic.go:356] "Generic (PLEG): container finished" podID="da49a5be-afe7-47a7-a0a5-5f05ca7d9231" containerID="b639292af1a02a12d4f09ca3a8408cdc737a73c6ae0621c068b2cf140cfee4e6" exitCode=0
Nov 03 21:43:33.976340 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:33.975922    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fff189-qsdv2" event={"ID":"da49a5be-afe7-47a7-a0a5-5f05ca7d9231","Type":"ContainerDied","Data":"b639292af1a02a12d4f09ca3a8408cdc737a73c6ae0621c068b2cf140cfee4e6"}
Nov 03 21:43:35.144213 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:35.144189    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fff189-qsdv2"
Nov 03 21:43:35.259387 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:35.259226    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da49a5be-afe7-47a7-a0a5-5f05ca7d9231-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da49a5be-afe7-47a7-a0a5-5f05ca7d9231-maas-api-openshift-ca\") pod \"da49a5be-afe7-47a7-a0a5-5f05ca7d9231\" (UID: \"da49a5be-afe7-47a7-a0a5-5f05ca7d9231\") "
Nov 03 21:43:35.259387 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:35.259312    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da49a5be-afe7-47a7-a0a5-5f05ca7d9231-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da49a5be-afe7-47a7-a0a5-5f05ca7d9231-maas-api-xks-ca\") pod \"da49a5be-afe7-47a7-a0a5-5f05ca7d9231\" (UID: \"da49a5be-afe7-47a7-a0a5-5f05ca7d9231\") "
Nov 03 21:43:35.259387 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:35.259379    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da49a5be-afe7-47a7-a0a5-5f05ca7d9231-kube-api-access-qqwf8\" (UniqueName: \"kubernetes.io/projected/da49a5be-afe7-47a7-a0a5-5f05ca7d9231-kube-api-access-qqwf8\") pod \"da49a5be-afe7-47a7-a0a5-5f05ca7d9231\" (UID: \"da49a5be-afe7-47a7-a0a5-5f05ca7d9231\") "
Nov 03 21:43:35.260108 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:35.260077    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da49a5be-afe7-47a7-a0a5-5f05ca7d9231-maas-api-openshift-ca" pod "da49a5be-afe7-47a7-a0a5-5f05ca7d9231" (UID: "da49a5be-afe7-47a7-a0a5-5f05ca7d9231"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:43:35.261316 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:35.261217    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da49a5be-afe7-47a7-a0a5-5f05ca7d9231-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:43:35.262758 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:35.262546    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da49a5be-afe7-47a7-a0a5-5f05ca7d9231-maas-api-xks-ca" pod "da49a5be-afe7-47a7-a0a5-5f05ca7d9231" (UID: "da49a5be-afe7-47a7-a0a5-5f05ca7d9231"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:43:35.262758 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:35.262709    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da49a5be-afe7-47a7-a0a5-5f05ca7d9231-kube-api-access-qqwf8" pod "da49a5be-afe7-47a7-a0a5-5f05ca7d9231" (UID: "da49a5be-afe7-47a7-a0a5-5f05ca7d9231"). InnerVolumeSpecName "kube-api-access-qqwf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:43:35.362596 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:35.362562    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da49a5be-afe7-47a7-a0a5-5f05ca7d9231-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:43:35.362596 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:35.362590    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqwf8\" (UniqueName: \"kubernetes.io/projected/da49a5be-afe7-47a7-a0a5-5f05ca7d9231-kube-api-access-qqwf8\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:43:35.990714 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:35.990672    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fff189-qsdv2" event={"ID":"da49a5be-afe7-47a7-a0a5-5f05ca7d9231","Type":"ContainerDied","Data":"eff3e5f24f1f17025e5e3821ab3708bc04808b6855b8fedb887f98353323ae76"}
Nov 03 21:43:35.990714 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:35.990703    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fff189-qsdv2"
Nov 03 21:43:35.990935 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:35.990729    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eff3e5f24f1f17025e5e3821ab3708bc04808b6855b8fedb887f98353323ae76"
Nov 03 21:43:40.929977 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:43:40.929931    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:43:46.965180 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:46.965148    5053 scope.go:122] "RemoveContainer" containerID="e0b071eb3568c72fc00f2226d626ca82304e50a60a5f38e16c41607e2b1bcbce"
Nov 03 21:43:46.976329 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:46.976309    5053 scope.go:122] "RemoveContainer" containerID="c8b33489f62be017180f12303f6539dc7e80ca49b4dd77d33dd3ab1a33d5d0d6"
Nov 03 21:43:46.986951 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:46.986933    5053 scope.go:122] "RemoveContainer" containerID="f3d042f71bb4a3a64071dab7577ce8575934493d2f020204b023cb581af05365"
Nov 03 21:43:50.423576 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:50.422374    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30f143-bqhhl"]
Nov 03 21:43:50.491861 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:50.491821    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30f143-bqhhl"]
Nov 03 21:43:51.667996 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:51.667965    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f4f30b0-e837-46ff-8df0-ec486b0c873d" path="/var/lib/kubelet/pods/5f4f30b0-e837-46ff-8df0-ec486b0c873d/volumes"
Nov 03 21:43:57.981672 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:57.981637    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8"]
Nov 03 21:43:57.982316 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:57.982301    5053 cpu_manager.go:414] "containerMap: removing container" podUID="da49a5be-afe7-47a7-a0a5-5f05ca7d9231" containerName="revoke-keys"
Nov 03 21:43:57.982316 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:57.982316    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da49a5be-afe7-47a7-a0a5-5f05ca7d9231" containerName="revoke-keys"
Nov 03 21:43:57.982459 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:57.982447    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="da49a5be-afe7-47a7-a0a5-5f05ca7d9231" containerName="revoke-keys"
Nov 03 21:43:57.987538 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:57.987517    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8"
Nov 03 21:43:58.042266 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:58.042199    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8"]
Nov 03 21:43:58.086768 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:58.086740    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:43:58.086768 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:58.086748    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:43:58.089566 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:58.089528    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnv7m\" (UniqueName: \"kubernetes.io/projected/812811a5-95b8-4618-b7f1-2b807477a790-kube-api-access-vnv7m\") pod \"maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8\" (UID: \"812811a5-95b8-4618-b7f1-2b807477a790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8"
Nov 03 21:43:58.089698 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:58.089586    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/812811a5-95b8-4618-b7f1-2b807477a790-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8\" (UID: \"812811a5-95b8-4618-b7f1-2b807477a790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8"
Nov 03 21:43:58.089834 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:58.089806    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/812811a5-95b8-4618-b7f1-2b807477a790-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8\" (UID: \"812811a5-95b8-4618-b7f1-2b807477a790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8"
Nov 03 21:43:58.190947 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:58.190909    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/812811a5-95b8-4618-b7f1-2b807477a790-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8\" (UID: \"812811a5-95b8-4618-b7f1-2b807477a790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8"
Nov 03 21:43:58.191148 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:58.191128    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnv7m\" (UniqueName: \"kubernetes.io/projected/812811a5-95b8-4618-b7f1-2b807477a790-kube-api-access-vnv7m\") pod \"maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8\" (UID: \"812811a5-95b8-4618-b7f1-2b807477a790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8"
Nov 03 21:43:58.191191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:58.191180    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/812811a5-95b8-4618-b7f1-2b807477a790-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8\" (UID: \"812811a5-95b8-4618-b7f1-2b807477a790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8"
Nov 03 21:43:58.191824 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:58.191801    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/812811a5-95b8-4618-b7f1-2b807477a790-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8\" (UID: \"812811a5-95b8-4618-b7f1-2b807477a790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8"
Nov 03 21:43:58.193699 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:58.193669    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/812811a5-95b8-4618-b7f1-2b807477a790-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8\" (UID: \"812811a5-95b8-4618-b7f1-2b807477a790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8"
Nov 03 21:43:58.247740 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:58.247667    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnv7m\" (UniqueName: \"kubernetes.io/projected/812811a5-95b8-4618-b7f1-2b807477a790-kube-api-access-vnv7m\") pod \"maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8\" (UID: \"812811a5-95b8-4618-b7f1-2b807477a790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8"
Nov 03 21:43:58.301442 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:58.301405    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8"
Nov 03 21:43:58.683635 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:58.683606    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8"]
Nov 03 21:43:58.684255 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:43:58.684220    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod812811a5_95b8_4618_b7f1_2b807477a790.slice/crio-d53b07083f828552c1ecded3ca730f60a3a5611d597c7572f3d367043f8142a6 WatchSource:0}: Error finding container d53b07083f828552c1ecded3ca730f60a3a5611d597c7572f3d367043f8142a6: Status 404 returned error can't find the container with id d53b07083f828552c1ecded3ca730f60a3a5611d597c7572f3d367043f8142a6
Nov 03 21:43:59.191572 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:59.191536    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8" event={"ID":"812811a5-95b8-4618-b7f1-2b807477a790","Type":"ContainerStarted","Data":"4d835e2cfa89e43c4073100db85d2b585533efd51a2dbf0229e58bdb41c37792"}
Nov 03 21:43:59.191958 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:59.191585    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8" event={"ID":"812811a5-95b8-4618-b7f1-2b807477a790","Type":"ContainerStarted","Data":"d53b07083f828552c1ecded3ca730f60a3a5611d597c7572f3d367043f8142a6"}
Nov 03 21:43:59.305864 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:43:59.305770    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8" podStartSLOduration=2.305756948 podStartE2EDuration="2.305756948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:43:59.303466671 +0000 UTC m=+1753.797388545" watchObservedRunningTime="2026-11-03 21:43:59.305756948 +0000 UTC m=+1753.799678854"
Nov 03 21:44:00.200800 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:00.200763    5053 generic.go:356] "Generic (PLEG): container finished" podID="812811a5-95b8-4618-b7f1-2b807477a790" containerID="4d835e2cfa89e43c4073100db85d2b585533efd51a2dbf0229e58bdb41c37792" exitCode=0
Nov 03 21:44:00.201201 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:00.200813    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8" event={"ID":"812811a5-95b8-4618-b7f1-2b807477a790","Type":"ContainerDied","Data":"4d835e2cfa89e43c4073100db85d2b585533efd51a2dbf0229e58bdb41c37792"}
Nov 03 21:44:01.378678 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:01.378652    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8"
Nov 03 21:44:01.537094 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:01.537008    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/812811a5-95b8-4618-b7f1-2b807477a790-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/812811a5-95b8-4618-b7f1-2b807477a790-maas-api-openshift-ca\") pod \"812811a5-95b8-4618-b7f1-2b807477a790\" (UID: \"812811a5-95b8-4618-b7f1-2b807477a790\") "
Nov 03 21:44:01.537252 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:01.537127    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/812811a5-95b8-4618-b7f1-2b807477a790-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/812811a5-95b8-4618-b7f1-2b807477a790-maas-api-xks-ca\") pod \"812811a5-95b8-4618-b7f1-2b807477a790\" (UID: \"812811a5-95b8-4618-b7f1-2b807477a790\") "
Nov 03 21:44:01.537252 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:01.537163    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/812811a5-95b8-4618-b7f1-2b807477a790-kube-api-access-vnv7m\" (UniqueName: \"kubernetes.io/projected/812811a5-95b8-4618-b7f1-2b807477a790-kube-api-access-vnv7m\") pod \"812811a5-95b8-4618-b7f1-2b807477a790\" (UID: \"812811a5-95b8-4618-b7f1-2b807477a790\") "
Nov 03 21:44:01.537516 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:01.537480    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/812811a5-95b8-4618-b7f1-2b807477a790-maas-api-openshift-ca" pod "812811a5-95b8-4618-b7f1-2b807477a790" (UID: "812811a5-95b8-4618-b7f1-2b807477a790"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:44:01.537828 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:01.537806    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/812811a5-95b8-4618-b7f1-2b807477a790-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:01.539443 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:01.539417    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/812811a5-95b8-4618-b7f1-2b807477a790-maas-api-xks-ca" pod "812811a5-95b8-4618-b7f1-2b807477a790" (UID: "812811a5-95b8-4618-b7f1-2b807477a790"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:44:01.539534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:01.539521    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/812811a5-95b8-4618-b7f1-2b807477a790-kube-api-access-vnv7m" pod "812811a5-95b8-4618-b7f1-2b807477a790" (UID: "812811a5-95b8-4618-b7f1-2b807477a790"). InnerVolumeSpecName "kube-api-access-vnv7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:01.639477 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:01.639440    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/812811a5-95b8-4618-b7f1-2b807477a790-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:01.639477 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:01.639470    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnv7m\" (UniqueName: \"kubernetes.io/projected/812811a5-95b8-4618-b7f1-2b807477a790-kube-api-access-vnv7m\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:02.221163 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:02.221125    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8"
Nov 03 21:44:02.221163 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:02.221136    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8" event={"ID":"812811a5-95b8-4618-b7f1-2b807477a790","Type":"ContainerDied","Data":"d53b07083f828552c1ecded3ca730f60a3a5611d597c7572f3d367043f8142a6"}
Nov 03 21:44:02.221412 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:02.221179    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d53b07083f828552c1ecded3ca730f60a3a5611d597c7572f3d367043f8142a6"
Nov 03 21:44:10.931877 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:44:10.931827    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:44:12.606205 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:12.606174    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"]
Nov 03 21:44:12.606697 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:12.606464    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9" podUID="91eb34d0-5de4-4a33-b596-0f69a14d4131" containerName="istio-proxy" containerID="cri-o://c5a7d4810f87ba57f6d4dc5490b9f557478e2330f5cd4e7632ff8b06fccfeccf" gracePeriod=30
Nov 03 21:44:12.622164 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:12.622130    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"]
Nov 03 21:44:12.622691 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:12.622662    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb" podUID="9b3caedf-8d81-4f57-b7b5-7d996b5a193a" containerName="istio-proxy" containerID="cri-o://929030f1cdeda8ff6b7c9210503a446de153892e3a81580f4046d10b40c9fce4" gracePeriod=30
Nov 03 21:44:12.626584 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:12.626556    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"]
Nov 03 21:44:12.626762 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:12.626737    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72" podUID="a3694a1a-4c11-45b5-aed6-2568770e962c" containerName="istio-proxy" containerID="cri-o://466c43ebd59361ffdfd74d1b6f6f33c5d6cb7087ec1535d21cef1254be173dd5" gracePeriod=30
Nov 03 21:44:12.656422 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:12.656387    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"]
Nov 03 21:44:12.656588 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:12.656570    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2" podUID="bdeb5c9c-5820-476c-9763-5ccad90f0c5d" containerName="istio-proxy" containerID="cri-o://41078f5752bd77d5f41000cca4db67ea60a0da238e3c87936e41a9dcb8019c8f" gracePeriod=30
Nov 03 21:44:16.045881 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:16.045845    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de3a73dd-bz6x9"]
Nov 03 21:44:16.095589 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:16.095561    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de3a73dd-bz6x9"]
Nov 03 21:44:16.762071 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:16.762033    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j"]
Nov 03 21:44:16.813093 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:16.813053    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5733b4-qqr4j"]
Nov 03 21:44:17.670015 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:17.669976    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f" path="/var/lib/kubelet/pods/1548f1f5-6ebc-49b2-8eb7-73d2c169cb2f/volumes"
Nov 03 21:44:17.670648 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:17.670630    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b576e48b-2608-4984-8630-4f449cdbe18a" path="/var/lib/kubelet/pods/b576e48b-2608-4984-8630-4f449cdbe18a/volumes"
Nov 03 21:44:17.993908 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:17.992963    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:44:18.000348 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.000321    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:44:18.059610 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.059586    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:44:18.063153 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.063127    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:44:18.077861 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.077837    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-envoy\") pod \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") "
Nov 03 21:44:18.077976 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.077892    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-workload-socket\") pod \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") "
Nov 03 21:44:18.077976 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.077955    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-podinfo\") pod \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") "
Nov 03 21:44:18.078112 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.078007    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-data\") pod \"91eb34d0-5de4-4a33-b596-0f69a14d4131\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") "
Nov 03 21:44:18.078112 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.078067    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-envoy\") pod \"91eb34d0-5de4-4a33-b596-0f69a14d4131\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") "
Nov 03 21:44:18.078221 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.078126    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-workload-socket\") pod \"91eb34d0-5de4-4a33-b596-0f69a14d4131\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") "
Nov 03 21:44:18.078913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.078221    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-workload-certs\") pod \"91eb34d0-5de4-4a33-b596-0f69a14d4131\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") "
Nov 03 21:44:18.078913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.078306    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-credential-socket\") pod \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") "
Nov 03 21:44:18.078913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.078339    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-kube-api-access-9dzpr\" (UniqueName: \"kubernetes.io/projected/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-kube-api-access-9dzpr\") pod \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") "
Nov 03 21:44:18.078913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.078381    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-credential-socket\") pod \"91eb34d0-5de4-4a33-b596-0f69a14d4131\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") "
Nov 03 21:44:18.078913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.078421    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istiod-ca-cert\") pod \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") "
Nov 03 21:44:18.078913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.078472    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-token\" (UniqueName: \"kubernetes.io/projected/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-token\") pod \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") "
Nov 03 21:44:18.078913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.078522    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91eb34d0-5de4-4a33-b596-0f69a14d4131-kube-api-access-drkzd\" (UniqueName: \"kubernetes.io/projected/91eb34d0-5de4-4a33-b596-0f69a14d4131-kube-api-access-drkzd\") pod \"91eb34d0-5de4-4a33-b596-0f69a14d4131\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") "
Nov 03 21:44:18.078913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.078523    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-workload-socket" pod "9b3caedf-8d81-4f57-b7b5-7d996b5a193a" (UID: "9b3caedf-8d81-4f57-b7b5-7d996b5a193a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.078913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.078557    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91eb34d0-5de4-4a33-b596-0f69a14d4131-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91eb34d0-5de4-4a33-b596-0f69a14d4131-istiod-ca-cert\") pod \"91eb34d0-5de4-4a33-b596-0f69a14d4131\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") "
Nov 03 21:44:18.078913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.078597    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-podinfo\") pod \"91eb34d0-5de4-4a33-b596-0f69a14d4131\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") "
Nov 03 21:44:18.078913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.078635    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-workload-certs\") pod \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") "
Nov 03 21:44:18.078913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.078693    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-token\" (UniqueName: \"kubernetes.io/projected/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-token\") pod \"91eb34d0-5de4-4a33-b596-0f69a14d4131\" (UID: \"91eb34d0-5de4-4a33-b596-0f69a14d4131\") "
Nov 03 21:44:18.078913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.078764    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-data\") pod \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\" (UID: \"9b3caedf-8d81-4f57-b7b5-7d996b5a193a\") "
Nov 03 21:44:18.078913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.078783    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-workload-certs" pod "91eb34d0-5de4-4a33-b596-0f69a14d4131" (UID: "91eb34d0-5de4-4a33-b596-0f69a14d4131"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.079651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.078962    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-workload-socket" pod "91eb34d0-5de4-4a33-b596-0f69a14d4131" (UID: "91eb34d0-5de4-4a33-b596-0f69a14d4131"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.079651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.078890    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-data" pod "91eb34d0-5de4-4a33-b596-0f69a14d4131" (UID: "91eb34d0-5de4-4a33-b596-0f69a14d4131"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.079651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.079119    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-credential-socket" pod "9b3caedf-8d81-4f57-b7b5-7d996b5a193a" (UID: "9b3caedf-8d81-4f57-b7b5-7d996b5a193a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.079651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.079335    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-credential-socket" pod "91eb34d0-5de4-4a33-b596-0f69a14d4131" (UID: "91eb34d0-5de4-4a33-b596-0f69a14d4131"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.079651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.079483    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.079651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.079504    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.079651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.079517    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.079651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.079532    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.079651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.079545    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.079651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.079559    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.080132 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.079858    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-workload-certs" pod "9b3caedf-8d81-4f57-b7b5-7d996b5a193a" (UID: "9b3caedf-8d81-4f57-b7b5-7d996b5a193a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.080132 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.079901    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istiod-ca-cert" pod "9b3caedf-8d81-4f57-b7b5-7d996b5a193a" (UID: "9b3caedf-8d81-4f57-b7b5-7d996b5a193a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:44:18.080259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.080155    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91eb34d0-5de4-4a33-b596-0f69a14d4131-istiod-ca-cert" pod "91eb34d0-5de4-4a33-b596-0f69a14d4131" (UID: "91eb34d0-5de4-4a33-b596-0f69a14d4131"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:44:18.081052 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.081022    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-envoy" pod "9b3caedf-8d81-4f57-b7b5-7d996b5a193a" (UID: "9b3caedf-8d81-4f57-b7b5-7d996b5a193a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.081154 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.081108    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-podinfo" pod "9b3caedf-8d81-4f57-b7b5-7d996b5a193a" (UID: "9b3caedf-8d81-4f57-b7b5-7d996b5a193a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:44:18.081279 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.081213    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-envoy" pod "91eb34d0-5de4-4a33-b596-0f69a14d4131" (UID: "91eb34d0-5de4-4a33-b596-0f69a14d4131"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.081406 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.081280    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-data" pod "9b3caedf-8d81-4f57-b7b5-7d996b5a193a" (UID: "9b3caedf-8d81-4f57-b7b5-7d996b5a193a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.082586 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.082563    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-token" pod "9b3caedf-8d81-4f57-b7b5-7d996b5a193a" (UID: "9b3caedf-8d81-4f57-b7b5-7d996b5a193a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:18.082673 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.082609    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91eb34d0-5de4-4a33-b596-0f69a14d4131-kube-api-access-drkzd" pod "91eb34d0-5de4-4a33-b596-0f69a14d4131" (UID: "91eb34d0-5de4-4a33-b596-0f69a14d4131"). InnerVolumeSpecName "kube-api-access-drkzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:18.082673 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.082634    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-kube-api-access-9dzpr" pod "9b3caedf-8d81-4f57-b7b5-7d996b5a193a" (UID: "9b3caedf-8d81-4f57-b7b5-7d996b5a193a"). InnerVolumeSpecName "kube-api-access-9dzpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:18.082915 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.082892    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-token" pod "91eb34d0-5de4-4a33-b596-0f69a14d4131" (UID: "91eb34d0-5de4-4a33-b596-0f69a14d4131"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:18.083921 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.083904    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-podinfo" pod "91eb34d0-5de4-4a33-b596-0f69a14d4131" (UID: "91eb34d0-5de4-4a33-b596-0f69a14d4131"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:44:18.181356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.180137    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-token\" (UniqueName: \"kubernetes.io/projected/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-token\") pod \"a3694a1a-4c11-45b5-aed6-2568770e962c\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") "
Nov 03 21:44:18.181356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.180190    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-data\") pod \"a3694a1a-4c11-45b5-aed6-2568770e962c\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") "
Nov 03 21:44:18.181356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.180286    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-podinfo\") pod \"a3694a1a-4c11-45b5-aed6-2568770e962c\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") "
Nov 03 21:44:18.181356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.180326    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-workload-socket\") pod \"a3694a1a-4c11-45b5-aed6-2568770e962c\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") "
Nov 03 21:44:18.181356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.180363    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istiod-ca-cert\") pod \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") "
Nov 03 21:44:18.181356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.180391    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-envoy\") pod \"a3694a1a-4c11-45b5-aed6-2568770e962c\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") "
Nov 03 21:44:18.181356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.180417    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3694a1a-4c11-45b5-aed6-2568770e962c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3694a1a-4c11-45b5-aed6-2568770e962c-istiod-ca-cert\") pod \"a3694a1a-4c11-45b5-aed6-2568770e962c\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") "
Nov 03 21:44:18.181356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.180480    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-token\" (UniqueName: \"kubernetes.io/projected/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-token\") pod \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") "
Nov 03 21:44:18.181356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.180531    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-workload-socket\") pod \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") "
Nov 03 21:44:18.181356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.180560    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-envoy\") pod \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") "
Nov 03 21:44:18.181356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.180612    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-workload-certs\") pod \"a3694a1a-4c11-45b5-aed6-2568770e962c\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") "
Nov 03 21:44:18.181356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.180666    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-workload-certs\") pod \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") "
Nov 03 21:44:18.181356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.180695    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3694a1a-4c11-45b5-aed6-2568770e962c-kube-api-access-hmr6n\" (UniqueName: \"kubernetes.io/projected/a3694a1a-4c11-45b5-aed6-2568770e962c-kube-api-access-hmr6n\") pod \"a3694a1a-4c11-45b5-aed6-2568770e962c\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") "
Nov 03 21:44:18.181356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.180753    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-data\") pod \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") "
Nov 03 21:44:18.181356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.180797    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-podinfo\") pod \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") "
Nov 03 21:44:18.181356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.180853    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-kube-api-access-p2ngd\" (UniqueName: \"kubernetes.io/projected/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-kube-api-access-p2ngd\") pod \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") "
Nov 03 21:44:18.181356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.180912    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-credential-socket\") pod \"a3694a1a-4c11-45b5-aed6-2568770e962c\" (UID: \"a3694a1a-4c11-45b5-aed6-2568770e962c\") "
Nov 03 21:44:18.181356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.181103    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-workload-socket" pod "a3694a1a-4c11-45b5-aed6-2568770e962c" (UID: "a3694a1a-4c11-45b5-aed6-2568770e962c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.181356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.181143    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-credential-socket\") pod \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\" (UID: \"bdeb5c9c-5820-476c-9763-5ccad90f0c5d\") "
Nov 03 21:44:18.182583 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.181477    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istiod-ca-cert" pod "bdeb5c9c-5820-476c-9763-5ccad90f0c5d" (UID: "bdeb5c9c-5820-476c-9763-5ccad90f0c5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:44:18.182583 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.181724    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-data" pod "a3694a1a-4c11-45b5-aed6-2568770e962c" (UID: "a3694a1a-4c11-45b5-aed6-2568770e962c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.182583 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.182057    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3694a1a-4c11-45b5-aed6-2568770e962c-istiod-ca-cert" pod "a3694a1a-4c11-45b5-aed6-2568770e962c" (UID: "a3694a1a-4c11-45b5-aed6-2568770e962c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:44:18.182583 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.182343    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-data" pod "bdeb5c9c-5820-476c-9763-5ccad90f0c5d" (UID: "bdeb5c9c-5820-476c-9763-5ccad90f0c5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.183897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.182923    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-workload-certs" pod "a3694a1a-4c11-45b5-aed6-2568770e962c" (UID: "a3694a1a-4c11-45b5-aed6-2568770e962c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.183897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.183037    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-token" pod "a3694a1a-4c11-45b5-aed6-2568770e962c" (UID: "a3694a1a-4c11-45b5-aed6-2568770e962c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:18.183897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.183148    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-workload-certs" pod "bdeb5c9c-5820-476c-9763-5ccad90f0c5d" (UID: "bdeb5c9c-5820-476c-9763-5ccad90f0c5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.183897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.183361    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.183897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.183385    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.183897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.183399    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.183897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.183412    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.183897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.183425    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.183897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.183437    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3694a1a-4c11-45b5-aed6-2568770e962c-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.183897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.183450    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.183897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.183461    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.183897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.183474    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.183897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.183486    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.183897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.183499    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.183897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.183512    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dzpr\" (UniqueName: \"kubernetes.io/projected/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-kube-api-access-9dzpr\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.183897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.183527    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.183897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.183542    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.183897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.183622    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drkzd\" (UniqueName: \"kubernetes.io/projected/91eb34d0-5de4-4a33-b596-0f69a14d4131-kube-api-access-drkzd\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.183897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.183783    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91eb34d0-5de4-4a33-b596-0f69a14d4131-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.183897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.183801    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.183897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.183815    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91eb34d0-5de4-4a33-b596-0f69a14d4131-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.183897 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.183829    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.185113 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.183843    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b3caedf-8d81-4f57-b7b5-7d996b5a193a-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.185113 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.184371    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-credential-socket" pod "bdeb5c9c-5820-476c-9763-5ccad90f0c5d" (UID: "bdeb5c9c-5820-476c-9763-5ccad90f0c5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.185460 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.185270    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-credential-socket" pod "a3694a1a-4c11-45b5-aed6-2568770e962c" (UID: "a3694a1a-4c11-45b5-aed6-2568770e962c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.185606 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.185577    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-workload-socket" pod "bdeb5c9c-5820-476c-9763-5ccad90f0c5d" (UID: "bdeb5c9c-5820-476c-9763-5ccad90f0c5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.188192 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.188172    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-podinfo" pod "a3694a1a-4c11-45b5-aed6-2568770e962c" (UID: "a3694a1a-4c11-45b5-aed6-2568770e962c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:44:18.193203 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.190562    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-podinfo" pod "bdeb5c9c-5820-476c-9763-5ccad90f0c5d" (UID: "bdeb5c9c-5820-476c-9763-5ccad90f0c5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:44:18.193203 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.190669    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-envoy" pod "a3694a1a-4c11-45b5-aed6-2568770e962c" (UID: "a3694a1a-4c11-45b5-aed6-2568770e962c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.193203 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.190764    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-envoy" pod "bdeb5c9c-5820-476c-9763-5ccad90f0c5d" (UID: "bdeb5c9c-5820-476c-9763-5ccad90f0c5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.193203 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.190817    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3694a1a-4c11-45b5-aed6-2568770e962c-kube-api-access-hmr6n" pod "a3694a1a-4c11-45b5-aed6-2568770e962c" (UID: "a3694a1a-4c11-45b5-aed6-2568770e962c"). InnerVolumeSpecName "kube-api-access-hmr6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:18.193203 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.190929    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-kube-api-access-p2ngd" pod "bdeb5c9c-5820-476c-9763-5ccad90f0c5d" (UID: "bdeb5c9c-5820-476c-9763-5ccad90f0c5d"). InnerVolumeSpecName "kube-api-access-p2ngd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:18.193203 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.192605    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-token" pod "bdeb5c9c-5820-476c-9763-5ccad90f0c5d" (UID: "bdeb5c9c-5820-476c-9763-5ccad90f0c5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:18.285584 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.285545    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.285584 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.285583    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.285854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.285597    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmr6n\" (UniqueName: \"kubernetes.io/projected/a3694a1a-4c11-45b5-aed6-2568770e962c-kube-api-access-hmr6n\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.285854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.285611    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.285854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.285622    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2ngd\" (UniqueName: \"kubernetes.io/projected/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-kube-api-access-p2ngd\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.285854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.285634    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.285854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.285645    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.285854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.285657    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.285854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.285668    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3694a1a-4c11-45b5-aed6-2568770e962c-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.285854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.285679    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdeb5c9c-5820-476c-9763-5ccad90f0c5d-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.363664 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.363629    5053 generic.go:356] "Generic (PLEG): container finished" podID="bdeb5c9c-5820-476c-9763-5ccad90f0c5d" containerID="41078f5752bd77d5f41000cca4db67ea60a0da238e3c87936e41a9dcb8019c8f" exitCode=0
Nov 03 21:44:18.363849 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.363694    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"
Nov 03 21:44:18.363849 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.363719    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2" event={"ID":"bdeb5c9c-5820-476c-9763-5ccad90f0c5d","Type":"ContainerDied","Data":"41078f5752bd77d5f41000cca4db67ea60a0da238e3c87936e41a9dcb8019c8f"}
Nov 03 21:44:18.363849 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.363784    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2" event={"ID":"bdeb5c9c-5820-476c-9763-5ccad90f0c5d","Type":"ContainerDied","Data":"7e37470aa296acf9c962e4c0b430bd63c28d38e64e2e27acb42bd294bcf67bae"}
Nov 03 21:44:18.363849 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.363812    5053 scope.go:122] "RemoveContainer" containerID="41078f5752bd77d5f41000cca4db67ea60a0da238e3c87936e41a9dcb8019c8f"
Nov 03 21:44:18.365506 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.365479    5053 generic.go:356] "Generic (PLEG): container finished" podID="a3694a1a-4c11-45b5-aed6-2568770e962c" containerID="466c43ebd59361ffdfd74d1b6f6f33c5d6cb7087ec1535d21cef1254be173dd5" exitCode=0
Nov 03 21:44:18.365653 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.365623    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72" event={"ID":"a3694a1a-4c11-45b5-aed6-2568770e962c","Type":"ContainerDied","Data":"466c43ebd59361ffdfd74d1b6f6f33c5d6cb7087ec1535d21cef1254be173dd5"}
Nov 03 21:44:18.365776 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.365684    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72" event={"ID":"a3694a1a-4c11-45b5-aed6-2568770e962c","Type":"ContainerDied","Data":"35f43826d7946f7b1a43d389cf82f162b7c02a88e0abe4c58fa1ddea839055bd"}
Nov 03 21:44:18.365776 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.365755    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"
Nov 03 21:44:18.369416 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.369391    5053 generic.go:356] "Generic (PLEG): container finished" podID="9b3caedf-8d81-4f57-b7b5-7d996b5a193a" containerID="929030f1cdeda8ff6b7c9210503a446de153892e3a81580f4046d10b40c9fce4" exitCode=0
Nov 03 21:44:18.369528 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.369430    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"
Nov 03 21:44:18.369528 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.369478    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb" event={"ID":"9b3caedf-8d81-4f57-b7b5-7d996b5a193a","Type":"ContainerDied","Data":"929030f1cdeda8ff6b7c9210503a446de153892e3a81580f4046d10b40c9fce4"}
Nov 03 21:44:18.369528 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.369522    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb" event={"ID":"9b3caedf-8d81-4f57-b7b5-7d996b5a193a","Type":"ContainerDied","Data":"6eaf40ebe1e8366807a9613251109783732295cdfa03a0732b1335934f36b027"}
Nov 03 21:44:18.371317 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.371294    5053 generic.go:356] "Generic (PLEG): container finished" podID="91eb34d0-5de4-4a33-b596-0f69a14d4131" containerID="c5a7d4810f87ba57f6d4dc5490b9f557478e2330f5cd4e7632ff8b06fccfeccf" exitCode=0
Nov 03 21:44:18.371428 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.371326    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9" event={"ID":"91eb34d0-5de4-4a33-b596-0f69a14d4131","Type":"ContainerDied","Data":"c5a7d4810f87ba57f6d4dc5490b9f557478e2330f5cd4e7632ff8b06fccfeccf"}
Nov 03 21:44:18.371428 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.371360    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9" event={"ID":"91eb34d0-5de4-4a33-b596-0f69a14d4131","Type":"ContainerDied","Data":"50bd617d4797db6c9fe6fe0f1481775bb68c8dfcac79c2752aaf204a9a7e941d"}
Nov 03 21:44:18.371428 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.371415    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"
Nov 03 21:44:18.377979 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.377964    5053 scope.go:122] "RemoveContainer" containerID="41078f5752bd77d5f41000cca4db67ea60a0da238e3c87936e41a9dcb8019c8f"
Nov 03 21:44:18.385178 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:44:18.385156    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41078f5752bd77d5f41000cca4db67ea60a0da238e3c87936e41a9dcb8019c8f\": container with ID starting with 41078f5752bd77d5f41000cca4db67ea60a0da238e3c87936e41a9dcb8019c8f not found: ID does not exist" containerID="41078f5752bd77d5f41000cca4db67ea60a0da238e3c87936e41a9dcb8019c8f"
Nov 03 21:44:18.385319 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.385184    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41078f5752bd77d5f41000cca4db67ea60a0da238e3c87936e41a9dcb8019c8f"} err="failed to get container status \"41078f5752bd77d5f41000cca4db67ea60a0da238e3c87936e41a9dcb8019c8f\": rpc error: code = NotFound desc = could not find container \"41078f5752bd77d5f41000cca4db67ea60a0da238e3c87936e41a9dcb8019c8f\": container with ID starting with 41078f5752bd77d5f41000cca4db67ea60a0da238e3c87936e41a9dcb8019c8f not found: ID does not exist"
Nov 03 21:44:18.385319 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.385199    5053 scope.go:122] "RemoveContainer" containerID="466c43ebd59361ffdfd74d1b6f6f33c5d6cb7087ec1535d21cef1254be173dd5"
Nov 03 21:44:18.395316 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.395282    5053 scope.go:122] "RemoveContainer" containerID="466c43ebd59361ffdfd74d1b6f6f33c5d6cb7087ec1535d21cef1254be173dd5"
Nov 03 21:44:18.396717 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:44:18.396285    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"466c43ebd59361ffdfd74d1b6f6f33c5d6cb7087ec1535d21cef1254be173dd5\": container with ID starting with 466c43ebd59361ffdfd74d1b6f6f33c5d6cb7087ec1535d21cef1254be173dd5 not found: ID does not exist" containerID="466c43ebd59361ffdfd74d1b6f6f33c5d6cb7087ec1535d21cef1254be173dd5"
Nov 03 21:44:18.396717 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.396345    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"466c43ebd59361ffdfd74d1b6f6f33c5d6cb7087ec1535d21cef1254be173dd5"} err="failed to get container status \"466c43ebd59361ffdfd74d1b6f6f33c5d6cb7087ec1535d21cef1254be173dd5\": rpc error: code = NotFound desc = could not find container \"466c43ebd59361ffdfd74d1b6f6f33c5d6cb7087ec1535d21cef1254be173dd5\": container with ID starting with 466c43ebd59361ffdfd74d1b6f6f33c5d6cb7087ec1535d21cef1254be173dd5 not found: ID does not exist"
Nov 03 21:44:18.396717 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.396638    5053 scope.go:122] "RemoveContainer" containerID="929030f1cdeda8ff6b7c9210503a446de153892e3a81580f4046d10b40c9fce4"
Nov 03 21:44:18.407384 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.407361    5053 scope.go:122] "RemoveContainer" containerID="098e22c3c107fac956d84a8e7fd019ca523dd73f358b2b276124e5a9b8e609e9"
Nov 03 21:44:18.419929 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.419896    5053 scope.go:122] "RemoveContainer" containerID="929030f1cdeda8ff6b7c9210503a446de153892e3a81580f4046d10b40c9fce4"
Nov 03 21:44:18.420219 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:44:18.420200    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"929030f1cdeda8ff6b7c9210503a446de153892e3a81580f4046d10b40c9fce4\": container with ID starting with 929030f1cdeda8ff6b7c9210503a446de153892e3a81580f4046d10b40c9fce4 not found: ID does not exist" containerID="929030f1cdeda8ff6b7c9210503a446de153892e3a81580f4046d10b40c9fce4"
Nov 03 21:44:18.420336 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.420227    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"929030f1cdeda8ff6b7c9210503a446de153892e3a81580f4046d10b40c9fce4"} err="failed to get container status \"929030f1cdeda8ff6b7c9210503a446de153892e3a81580f4046d10b40c9fce4\": rpc error: code = NotFound desc = could not find container \"929030f1cdeda8ff6b7c9210503a446de153892e3a81580f4046d10b40c9fce4\": container with ID starting with 929030f1cdeda8ff6b7c9210503a446de153892e3a81580f4046d10b40c9fce4 not found: ID does not exist"
Nov 03 21:44:18.420336 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.420265    5053 scope.go:122] "RemoveContainer" containerID="098e22c3c107fac956d84a8e7fd019ca523dd73f358b2b276124e5a9b8e609e9"
Nov 03 21:44:18.420547 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:44:18.420532    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"098e22c3c107fac956d84a8e7fd019ca523dd73f358b2b276124e5a9b8e609e9\": container with ID starting with 098e22c3c107fac956d84a8e7fd019ca523dd73f358b2b276124e5a9b8e609e9 not found: ID does not exist" containerID="098e22c3c107fac956d84a8e7fd019ca523dd73f358b2b276124e5a9b8e609e9"
Nov 03 21:44:18.420613 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.420552    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"098e22c3c107fac956d84a8e7fd019ca523dd73f358b2b276124e5a9b8e609e9"} err="failed to get container status \"098e22c3c107fac956d84a8e7fd019ca523dd73f358b2b276124e5a9b8e609e9\": rpc error: code = NotFound desc = could not find container \"098e22c3c107fac956d84a8e7fd019ca523dd73f358b2b276124e5a9b8e609e9\": container with ID starting with 098e22c3c107fac956d84a8e7fd019ca523dd73f358b2b276124e5a9b8e609e9 not found: ID does not exist"
Nov 03 21:44:18.420613 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.420567    5053 scope.go:122] "RemoveContainer" containerID="c5a7d4810f87ba57f6d4dc5490b9f557478e2330f5cd4e7632ff8b06fccfeccf"
Nov 03 21:44:18.432676 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.432556    5053 scope.go:122] "RemoveContainer" containerID="c5a7d4810f87ba57f6d4dc5490b9f557478e2330f5cd4e7632ff8b06fccfeccf"
Nov 03 21:44:18.432889 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:44:18.432861    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5a7d4810f87ba57f6d4dc5490b9f557478e2330f5cd4e7632ff8b06fccfeccf\": container with ID starting with c5a7d4810f87ba57f6d4dc5490b9f557478e2330f5cd4e7632ff8b06fccfeccf not found: ID does not exist" containerID="c5a7d4810f87ba57f6d4dc5490b9f557478e2330f5cd4e7632ff8b06fccfeccf"
Nov 03 21:44:18.432992 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.432896    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a7d4810f87ba57f6d4dc5490b9f557478e2330f5cd4e7632ff8b06fccfeccf"} err="failed to get container status \"c5a7d4810f87ba57f6d4dc5490b9f557478e2330f5cd4e7632ff8b06fccfeccf\": rpc error: code = NotFound desc = could not find container \"c5a7d4810f87ba57f6d4dc5490b9f557478e2330f5cd4e7632ff8b06fccfeccf\": container with ID starting with c5a7d4810f87ba57f6d4dc5490b9f557478e2330f5cd4e7632ff8b06fccfeccf not found: ID does not exist"
Nov 03 21:44:18.626703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.626673    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"]
Nov 03 21:44:18.626869 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.626720    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-v7bp9"]
Nov 03 21:44:18.737082 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.737045    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"]
Nov 03 21:44:18.740838 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.740802    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-xjmlb"]
Nov 03 21:44:18.848385 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.847709    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"]
Nov 03 21:44:18.853802 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.853772    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-sxh72"]
Nov 03 21:44:18.962482 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.962442    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"]
Nov 03 21:44:18.965536 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:18.965512    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-p47p2"]
Nov 03 21:44:19.667653 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:19.667623    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91eb34d0-5de4-4a33-b596-0f69a14d4131" path="/var/lib/kubelet/pods/91eb34d0-5de4-4a33-b596-0f69a14d4131/volumes"
Nov 03 21:44:19.668103 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:19.668090    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b3caedf-8d81-4f57-b7b5-7d996b5a193a" path="/var/lib/kubelet/pods/9b3caedf-8d81-4f57-b7b5-7d996b5a193a/volumes"
Nov 03 21:44:19.668531 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:19.668518    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3694a1a-4c11-45b5-aed6-2568770e962c" path="/var/lib/kubelet/pods/a3694a1a-4c11-45b5-aed6-2568770e962c/volumes"
Nov 03 21:44:19.669251 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:19.669219    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdeb5c9c-5820-476c-9763-5ccad90f0c5d" path="/var/lib/kubelet/pods/bdeb5c9c-5820-476c-9763-5ccad90f0c5d/volumes"
Nov 03 21:44:32.811120 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:32.811078    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad161-m2jq8"]
Nov 03 21:44:32.858285 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:32.858223    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad161-m2jq8"]
Nov 03 21:44:33.667530 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:33.667498    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7" path="/var/lib/kubelet/pods/0f2d21eb-e40d-4d48-a8b8-39bf88f1ddf7/volumes"
Nov 03 21:44:39.140574 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:39.140527    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fff189-qsdv2"]
Nov 03 21:44:39.188535 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:39.188503    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fff189-qsdv2"]
Nov 03 21:44:39.667075 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:39.667047    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da49a5be-afe7-47a7-a0a5-5f05ca7d9231" path="/var/lib/kubelet/pods/da49a5be-afe7-47a7-a0a5-5f05ca7d9231/volumes"
Nov 03 21:44:40.933566 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:44:40.933525    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:44:44.238678 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.238637    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b9ed93-4rztg"]
Nov 03 21:44:44.239281 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.239259    5053 cpu_manager.go:414] "containerMap: removing container" podUID="a3694a1a-4c11-45b5-aed6-2568770e962c" containerName="istio-proxy"
Nov 03 21:44:44.239281 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.239278    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3694a1a-4c11-45b5-aed6-2568770e962c" containerName="istio-proxy"
Nov 03 21:44:44.239375 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.239303    5053 cpu_manager.go:414] "containerMap: removing container" podUID="91eb34d0-5de4-4a33-b596-0f69a14d4131" containerName="istio-proxy"
Nov 03 21:44:44.239375 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.239308    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91eb34d0-5de4-4a33-b596-0f69a14d4131" containerName="istio-proxy"
Nov 03 21:44:44.239375 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.239349    5053 cpu_manager.go:414] "containerMap: removing container" podUID="9b3caedf-8d81-4f57-b7b5-7d996b5a193a" containerName="istio-proxy"
Nov 03 21:44:44.239375 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.239354    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b3caedf-8d81-4f57-b7b5-7d996b5a193a" containerName="istio-proxy"
Nov 03 21:44:44.239375 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.239362    5053 cpu_manager.go:414] "containerMap: removing container" podUID="bdeb5c9c-5820-476c-9763-5ccad90f0c5d" containerName="istio-proxy"
Nov 03 21:44:44.239375 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.239367    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdeb5c9c-5820-476c-9763-5ccad90f0c5d" containerName="istio-proxy"
Nov 03 21:44:44.239561 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.239378    5053 cpu_manager.go:414] "containerMap: removing container" podUID="812811a5-95b8-4618-b7f1-2b807477a790" containerName="revoke-keys"
Nov 03 21:44:44.239561 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.239382    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="812811a5-95b8-4618-b7f1-2b807477a790" containerName="revoke-keys"
Nov 03 21:44:44.239561 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.239479    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b3caedf-8d81-4f57-b7b5-7d996b5a193a" containerName="istio-proxy"
Nov 03 21:44:44.239561 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.239489    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3694a1a-4c11-45b5-aed6-2568770e962c" containerName="istio-proxy"
Nov 03 21:44:44.239561 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.239496    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="91eb34d0-5de4-4a33-b596-0f69a14d4131" containerName="istio-proxy"
Nov 03 21:44:44.239561 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.239506    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b3caedf-8d81-4f57-b7b5-7d996b5a193a" containerName="istio-proxy"
Nov 03 21:44:44.239561 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.239513    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdeb5c9c-5820-476c-9763-5ccad90f0c5d" containerName="istio-proxy"
Nov 03 21:44:44.239561 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.239519    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="812811a5-95b8-4618-b7f1-2b807477a790" containerName="revoke-keys"
Nov 03 21:44:44.244071 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.244055    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9ed93-4rztg"
Nov 03 21:44:44.299107 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.299074    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b9ed93-4rztg"]
Nov 03 21:44:44.344896 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.344859    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:44:44.345095 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.344976    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:44:44.372082 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.372038    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61761e5-ce80-491c-b6f0-ed22df43ad19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28b9ed93-4rztg\" (UID: \"c61761e5-ce80-491c-b6f0-ed22df43ad19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9ed93-4rztg"
Nov 03 21:44:44.372288 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.372178    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61761e5-ce80-491c-b6f0-ed22df43ad19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28b9ed93-4rztg\" (UID: \"c61761e5-ce80-491c-b6f0-ed22df43ad19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9ed93-4rztg"
Nov 03 21:44:44.372288 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.372212    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sf6t\" (UniqueName: \"kubernetes.io/projected/c61761e5-ce80-491c-b6f0-ed22df43ad19-kube-api-access-9sf6t\") pod \"maas-api-revoke-sub-28b9ed93-4rztg\" (UID: \"c61761e5-ce80-491c-b6f0-ed22df43ad19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9ed93-4rztg"
Nov 03 21:44:44.473508 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.473471    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61761e5-ce80-491c-b6f0-ed22df43ad19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28b9ed93-4rztg\" (UID: \"c61761e5-ce80-491c-b6f0-ed22df43ad19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9ed93-4rztg"
Nov 03 21:44:44.473721 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.473572    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61761e5-ce80-491c-b6f0-ed22df43ad19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28b9ed93-4rztg\" (UID: \"c61761e5-ce80-491c-b6f0-ed22df43ad19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9ed93-4rztg"
Nov 03 21:44:44.473721 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.473603    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sf6t\" (UniqueName: \"kubernetes.io/projected/c61761e5-ce80-491c-b6f0-ed22df43ad19-kube-api-access-9sf6t\") pod \"maas-api-revoke-sub-28b9ed93-4rztg\" (UID: \"c61761e5-ce80-491c-b6f0-ed22df43ad19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9ed93-4rztg"
Nov 03 21:44:44.474156 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.474135    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61761e5-ce80-491c-b6f0-ed22df43ad19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28b9ed93-4rztg\" (UID: \"c61761e5-ce80-491c-b6f0-ed22df43ad19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9ed93-4rztg"
Nov 03 21:44:44.476012 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.475989    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61761e5-ce80-491c-b6f0-ed22df43ad19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28b9ed93-4rztg\" (UID: \"c61761e5-ce80-491c-b6f0-ed22df43ad19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9ed93-4rztg"
Nov 03 21:44:44.531770 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.531695    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sf6t\" (UniqueName: \"kubernetes.io/projected/c61761e5-ce80-491c-b6f0-ed22df43ad19-kube-api-access-9sf6t\") pod \"maas-api-revoke-sub-28b9ed93-4rztg\" (UID: \"c61761e5-ce80-491c-b6f0-ed22df43ad19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9ed93-4rztg"
Nov 03 21:44:44.558727 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.558698    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9ed93-4rztg"
Nov 03 21:44:44.938791 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:44.938700    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b9ed93-4rztg"]
Nov 03 21:44:44.940413 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:44:44.940385    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61761e5_ce80_491c_b6f0_ed22df43ad19.slice/crio-866a6bbcbea3606d748038e11a5c26e1b8b2b43d96ade47c93e208f3e8134f06 WatchSource:0}: Error finding container 866a6bbcbea3606d748038e11a5c26e1b8b2b43d96ade47c93e208f3e8134f06: Status 404 returned error can't find the container with id 866a6bbcbea3606d748038e11a5c26e1b8b2b43d96ade47c93e208f3e8134f06
Nov 03 21:44:45.583268 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:45.583116    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9ed93-4rztg" event={"ID":"c61761e5-ce80-491c-b6f0-ed22df43ad19","Type":"ContainerStarted","Data":"74fb0c452a2d30e92c789dbd05861ed72cda0a0fc766f8c141fc8383d839b938"}
Nov 03 21:44:45.583268 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:45.583187    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9ed93-4rztg" event={"ID":"c61761e5-ce80-491c-b6f0-ed22df43ad19","Type":"ContainerStarted","Data":"866a6bbcbea3606d748038e11a5c26e1b8b2b43d96ade47c93e208f3e8134f06"}
Nov 03 21:44:45.631035 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:45.630999    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"]
Nov 03 21:44:45.631246 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:45.631210    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq" podUID="83a53cc7-96e4-49db-a8ac-2d2de9e84188" containerName="istio-proxy" containerID="cri-o://20c18c028d05a29a3d45f681554e95cf34d5bbe9071132ec3225acad81c13c2c" gracePeriod=30
Nov 03 21:44:45.707881 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:45.707809    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9ed93-4rztg" podStartSLOduration=1.7077865700000001 podStartE2EDuration="1.70778657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:45.695309712 +0000 UTC m=+1800.189231588" watchObservedRunningTime="2026-11-03 21:44:45.70778657 +0000 UTC m=+1800.201708468"
Nov 03 21:44:45.944596 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:45.944566    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 21:44:45.953924 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:45.953899    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:44:45.962952 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:45.962929    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 21:44:45.970685 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:45.970664    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:44:46.470714 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:46.470678    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc"]
Nov 03 21:44:46.471358 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:46.471343    5053 cpu_manager.go:414] "containerMap: removing container" podUID="9b3caedf-8d81-4f57-b7b5-7d996b5a193a" containerName="istio-proxy"
Nov 03 21:44:46.471411 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:46.471359    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b3caedf-8d81-4f57-b7b5-7d996b5a193a" containerName="istio-proxy"
Nov 03 21:44:46.474849 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:46.474834    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc"
Nov 03 21:44:46.541805 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:46.541769    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc"]
Nov 03 21:44:46.593129 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:46.593094    5053 generic.go:356] "Generic (PLEG): container finished" podID="c61761e5-ce80-491c-b6f0-ed22df43ad19" containerID="74fb0c452a2d30e92c789dbd05861ed72cda0a0fc766f8c141fc8383d839b938" exitCode=0
Nov 03 21:44:46.593583 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:46.593151    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9ed93-4rztg" event={"ID":"c61761e5-ce80-491c-b6f0-ed22df43ad19","Type":"ContainerDied","Data":"74fb0c452a2d30e92c789dbd05861ed72cda0a0fc766f8c141fc8383d839b938"}
Nov 03 21:44:46.602306 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:46.602277    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7rcr\" (UniqueName: \"kubernetes.io/projected/d64bdf87-f4f6-4015-9d08-f7d4feeb18fc-kube-api-access-n7rcr\") pod \"maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc\" (UID: \"d64bdf87-f4f6-4015-9d08-f7d4feeb18fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc"
Nov 03 21:44:46.602483 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:46.602405    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64bdf87-f4f6-4015-9d08-f7d4feeb18fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc\" (UID: \"d64bdf87-f4f6-4015-9d08-f7d4feeb18fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc"
Nov 03 21:44:46.602543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:46.602481    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64bdf87-f4f6-4015-9d08-f7d4feeb18fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc\" (UID: \"d64bdf87-f4f6-4015-9d08-f7d4feeb18fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc"
Nov 03 21:44:46.704337 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:46.704300    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7rcr\" (UniqueName: \"kubernetes.io/projected/d64bdf87-f4f6-4015-9d08-f7d4feeb18fc-kube-api-access-n7rcr\") pod \"maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc\" (UID: \"d64bdf87-f4f6-4015-9d08-f7d4feeb18fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc"
Nov 03 21:44:46.704516 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:46.704377    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64bdf87-f4f6-4015-9d08-f7d4feeb18fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc\" (UID: \"d64bdf87-f4f6-4015-9d08-f7d4feeb18fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc"
Nov 03 21:44:46.704516 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:46.704441    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64bdf87-f4f6-4015-9d08-f7d4feeb18fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc\" (UID: \"d64bdf87-f4f6-4015-9d08-f7d4feeb18fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc"
Nov 03 21:44:46.705225 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:46.705156    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64bdf87-f4f6-4015-9d08-f7d4feeb18fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc\" (UID: \"d64bdf87-f4f6-4015-9d08-f7d4feeb18fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc"
Nov 03 21:44:46.707033 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:46.707009    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64bdf87-f4f6-4015-9d08-f7d4feeb18fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc\" (UID: \"d64bdf87-f4f6-4015-9d08-f7d4feeb18fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc"
Nov 03 21:44:46.761126 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:46.761060    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7rcr\" (UniqueName: \"kubernetes.io/projected/d64bdf87-f4f6-4015-9d08-f7d4feeb18fc-kube-api-access-n7rcr\") pod \"maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc\" (UID: \"d64bdf87-f4f6-4015-9d08-f7d4feeb18fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc"
Nov 03 21:44:46.788901 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:46.788875    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc"
Nov 03 21:44:47.170752 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:47.170719    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc"]
Nov 03 21:44:47.172112 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:44:47.172084    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64bdf87_f4f6_4015_9d08_f7d4feeb18fc.slice/crio-d1b62c8967f689bd62a93556fbf8bfd36803e0cac873188cfb46fd984c85ba9f WatchSource:0}: Error finding container d1b62c8967f689bd62a93556fbf8bfd36803e0cac873188cfb46fd984c85ba9f: Status 404 returned error can't find the container with id d1b62c8967f689bd62a93556fbf8bfd36803e0cac873188cfb46fd984c85ba9f
Nov 03 21:44:47.602692 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:47.602655    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc" event={"ID":"d64bdf87-f4f6-4015-9d08-f7d4feeb18fc","Type":"ContainerStarted","Data":"6bb82cb57fea8944136fa9288435fe7c05eef3cdb53f10074ea8aa38a9fdc1c2"}
Nov 03 21:44:47.603162 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:47.602714    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc" event={"ID":"d64bdf87-f4f6-4015-9d08-f7d4feeb18fc","Type":"ContainerStarted","Data":"d1b62c8967f689bd62a93556fbf8bfd36803e0cac873188cfb46fd984c85ba9f"}
Nov 03 21:44:47.733294 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:47.726721    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc" podStartSLOduration=1.7266999250000001 podStartE2EDuration="1.726699925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:47.715423636 +0000 UTC m=+1802.209345513" watchObservedRunningTime="2026-11-03 21:44:47.726699925 +0000 UTC m=+1802.220621782"
Nov 03 21:44:47.832133 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:47.832107    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9ed93-4rztg"
Nov 03 21:44:47.922671 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:47.922630    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c61761e5-ce80-491c-b6f0-ed22df43ad19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61761e5-ce80-491c-b6f0-ed22df43ad19-maas-api-openshift-ca\") pod \"c61761e5-ce80-491c-b6f0-ed22df43ad19\" (UID: \"c61761e5-ce80-491c-b6f0-ed22df43ad19\") "
Nov 03 21:44:47.922871 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:47.922689    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c61761e5-ce80-491c-b6f0-ed22df43ad19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61761e5-ce80-491c-b6f0-ed22df43ad19-maas-api-xks-ca\") pod \"c61761e5-ce80-491c-b6f0-ed22df43ad19\" (UID: \"c61761e5-ce80-491c-b6f0-ed22df43ad19\") "
Nov 03 21:44:47.922871 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:47.922804    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c61761e5-ce80-491c-b6f0-ed22df43ad19-kube-api-access-9sf6t\" (UniqueName: \"kubernetes.io/projected/c61761e5-ce80-491c-b6f0-ed22df43ad19-kube-api-access-9sf6t\") pod \"c61761e5-ce80-491c-b6f0-ed22df43ad19\" (UID: \"c61761e5-ce80-491c-b6f0-ed22df43ad19\") "
Nov 03 21:44:47.923070 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:47.923041    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c61761e5-ce80-491c-b6f0-ed22df43ad19-maas-api-openshift-ca" pod "c61761e5-ce80-491c-b6f0-ed22df43ad19" (UID: "c61761e5-ce80-491c-b6f0-ed22df43ad19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:44:47.923496 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:47.923468    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61761e5-ce80-491c-b6f0-ed22df43ad19-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:47.924951 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:47.924927    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c61761e5-ce80-491c-b6f0-ed22df43ad19-maas-api-xks-ca" pod "c61761e5-ce80-491c-b6f0-ed22df43ad19" (UID: "c61761e5-ce80-491c-b6f0-ed22df43ad19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:44:47.925057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:47.924970    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61761e5-ce80-491c-b6f0-ed22df43ad19-kube-api-access-9sf6t" pod "c61761e5-ce80-491c-b6f0-ed22df43ad19" (UID: "c61761e5-ce80-491c-b6f0-ed22df43ad19"). InnerVolumeSpecName "kube-api-access-9sf6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:48.025344 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:48.025210    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61761e5-ce80-491c-b6f0-ed22df43ad19-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:48.025344 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:48.025296    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sf6t\" (UniqueName: \"kubernetes.io/projected/c61761e5-ce80-491c-b6f0-ed22df43ad19-kube-api-access-9sf6t\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:48.611830 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:48.611793    5053 generic.go:356] "Generic (PLEG): container finished" podID="d64bdf87-f4f6-4015-9d08-f7d4feeb18fc" containerID="6bb82cb57fea8944136fa9288435fe7c05eef3cdb53f10074ea8aa38a9fdc1c2" exitCode=0
Nov 03 21:44:48.612297 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:48.611886    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc" event={"ID":"d64bdf87-f4f6-4015-9d08-f7d4feeb18fc","Type":"ContainerDied","Data":"6bb82cb57fea8944136fa9288435fe7c05eef3cdb53f10074ea8aa38a9fdc1c2"}
Nov 03 21:44:48.613351 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:48.613335    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9ed93-4rztg"
Nov 03 21:44:48.613458 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:48.613354    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9ed93-4rztg" event={"ID":"c61761e5-ce80-491c-b6f0-ed22df43ad19","Type":"ContainerDied","Data":"866a6bbcbea3606d748038e11a5c26e1b8b2b43d96ade47c93e208f3e8134f06"}
Nov 03 21:44:48.613458 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:48.613388    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="866a6bbcbea3606d748038e11a5c26e1b8b2b43d96ade47c93e208f3e8134f06"
Nov 03 21:44:49.789764 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:49.789740    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc"
Nov 03 21:44:49.949932 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:49.949896    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d64bdf87-f4f6-4015-9d08-f7d4feeb18fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64bdf87-f4f6-4015-9d08-f7d4feeb18fc-maas-api-xks-ca\") pod \"d64bdf87-f4f6-4015-9d08-f7d4feeb18fc\" (UID: \"d64bdf87-f4f6-4015-9d08-f7d4feeb18fc\") "
Nov 03 21:44:49.950164 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:49.950138    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d64bdf87-f4f6-4015-9d08-f7d4feeb18fc-kube-api-access-n7rcr\" (UniqueName: \"kubernetes.io/projected/d64bdf87-f4f6-4015-9d08-f7d4feeb18fc-kube-api-access-n7rcr\") pod \"d64bdf87-f4f6-4015-9d08-f7d4feeb18fc\" (UID: \"d64bdf87-f4f6-4015-9d08-f7d4feeb18fc\") "
Nov 03 21:44:49.950532 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:49.950179    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d64bdf87-f4f6-4015-9d08-f7d4feeb18fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64bdf87-f4f6-4015-9d08-f7d4feeb18fc-maas-api-openshift-ca\") pod \"d64bdf87-f4f6-4015-9d08-f7d4feeb18fc\" (UID: \"d64bdf87-f4f6-4015-9d08-f7d4feeb18fc\") "
Nov 03 21:44:49.951577 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:49.951472    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d64bdf87-f4f6-4015-9d08-f7d4feeb18fc-maas-api-openshift-ca" pod "d64bdf87-f4f6-4015-9d08-f7d4feeb18fc" (UID: "d64bdf87-f4f6-4015-9d08-f7d4feeb18fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:44:49.952219 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:49.952069    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64bdf87-f4f6-4015-9d08-f7d4feeb18fc-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:49.952876 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:49.952851    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d64bdf87-f4f6-4015-9d08-f7d4feeb18fc-maas-api-xks-ca" pod "d64bdf87-f4f6-4015-9d08-f7d4feeb18fc" (UID: "d64bdf87-f4f6-4015-9d08-f7d4feeb18fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:44:49.953131 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:49.953105    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d64bdf87-f4f6-4015-9d08-f7d4feeb18fc-kube-api-access-n7rcr" pod "d64bdf87-f4f6-4015-9d08-f7d4feeb18fc" (UID: "d64bdf87-f4f6-4015-9d08-f7d4feeb18fc"). InnerVolumeSpecName "kube-api-access-n7rcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:50.053560 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:50.053527    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7rcr\" (UniqueName: \"kubernetes.io/projected/d64bdf87-f4f6-4015-9d08-f7d4feeb18fc-kube-api-access-n7rcr\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:50.053560 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:50.053558    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64bdf87-f4f6-4015-9d08-f7d4feeb18fc-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:50.629348 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:50.629304    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc" event={"ID":"d64bdf87-f4f6-4015-9d08-f7d4feeb18fc","Type":"ContainerDied","Data":"d1b62c8967f689bd62a93556fbf8bfd36803e0cac873188cfb46fd984c85ba9f"}
Nov 03 21:44:50.629534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:50.629378    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1b62c8967f689bd62a93556fbf8bfd36803e0cac873188cfb46fd984c85ba9f"
Nov 03 21:44:50.629534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:50.629320    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc"
Nov 03 21:44:50.911659 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:50.911633    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:44:51.065713 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.065680    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-workload-socket\") pod \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") "
Nov 03 21:44:51.065925 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.065731    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-envoy\") pod \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") "
Nov 03 21:44:51.065925 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.065772    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-credential-socket\") pod \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") "
Nov 03 21:44:51.065925 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.065830    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-data\") pod \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") "
Nov 03 21:44:51.065925 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.065859    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-token\" (UniqueName: \"kubernetes.io/projected/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-token\") pod \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") "
Nov 03 21:44:51.065925 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.065922    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a53cc7-96e4-49db-a8ac-2d2de9e84188-kube-api-access-kbtbz\" (UniqueName: \"kubernetes.io/projected/83a53cc7-96e4-49db-a8ac-2d2de9e84188-kube-api-access-kbtbz\") pod \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") "
Nov 03 21:44:51.066203 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.065990    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-workload-certs\") pod \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") "
Nov 03 21:44:51.066203 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.066066    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-podinfo\") pod \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") "
Nov 03 21:44:51.066203 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.066100    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istiod-ca-cert\") pod \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\" (UID: \"83a53cc7-96e4-49db-a8ac-2d2de9e84188\") "
Nov 03 21:44:51.066203 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.066169    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-credential-socket" pod "83a53cc7-96e4-49db-a8ac-2d2de9e84188" (UID: "83a53cc7-96e4-49db-a8ac-2d2de9e84188"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:51.066896 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.066484    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istiod-ca-cert" pod "83a53cc7-96e4-49db-a8ac-2d2de9e84188" (UID: "83a53cc7-96e4-49db-a8ac-2d2de9e84188"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:44:51.066896 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.066711    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-workload-certs" pod "83a53cc7-96e4-49db-a8ac-2d2de9e84188" (UID: "83a53cc7-96e4-49db-a8ac-2d2de9e84188"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:51.067062 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.066932    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:51.067062 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.066952    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:51.067062 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.066964    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:51.067062 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.067010    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-workload-socket" pod "83a53cc7-96e4-49db-a8ac-2d2de9e84188" (UID: "83a53cc7-96e4-49db-a8ac-2d2de9e84188"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:51.067356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.067204    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-data" pod "83a53cc7-96e4-49db-a8ac-2d2de9e84188" (UID: "83a53cc7-96e4-49db-a8ac-2d2de9e84188"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:51.069556 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.069529    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-podinfo" pod "83a53cc7-96e4-49db-a8ac-2d2de9e84188" (UID: "83a53cc7-96e4-49db-a8ac-2d2de9e84188"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:44:51.069992 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.069968    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a53cc7-96e4-49db-a8ac-2d2de9e84188-kube-api-access-kbtbz" pod "83a53cc7-96e4-49db-a8ac-2d2de9e84188" (UID: "83a53cc7-96e4-49db-a8ac-2d2de9e84188"). InnerVolumeSpecName "kube-api-access-kbtbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:51.070070 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.069994    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-token" pod "83a53cc7-96e4-49db-a8ac-2d2de9e84188" (UID: "83a53cc7-96e4-49db-a8ac-2d2de9e84188"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:51.070127 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.070105    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-envoy" pod "83a53cc7-96e4-49db-a8ac-2d2de9e84188" (UID: "83a53cc7-96e4-49db-a8ac-2d2de9e84188"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:51.167912 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.167823    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:51.167912 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.167858    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:51.167912 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.167868    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbtbz\" (UniqueName: \"kubernetes.io/projected/83a53cc7-96e4-49db-a8ac-2d2de9e84188-kube-api-access-kbtbz\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:51.167912 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.167877    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:51.167912 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.167885    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:51.167912 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.167894    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a53cc7-96e4-49db-a8ac-2d2de9e84188-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:51.639052 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.639021    5053 generic.go:356] "Generic (PLEG): container finished" podID="83a53cc7-96e4-49db-a8ac-2d2de9e84188" containerID="20c18c028d05a29a3d45f681554e95cf34d5bbe9071132ec3225acad81c13c2c" exitCode=0
Nov 03 21:44:51.639266 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.639077    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"
Nov 03 21:44:51.639266 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.639110    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq" event={"ID":"83a53cc7-96e4-49db-a8ac-2d2de9e84188","Type":"ContainerDied","Data":"20c18c028d05a29a3d45f681554e95cf34d5bbe9071132ec3225acad81c13c2c"}
Nov 03 21:44:51.639266 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.639173    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq" event={"ID":"83a53cc7-96e4-49db-a8ac-2d2de9e84188","Type":"ContainerDied","Data":"7c46de45d621afbb5b5a6d0b8ba69a41609d0457d7d736cb54bb4828201a4e52"}
Nov 03 21:44:51.639266 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.639202    5053 scope.go:122] "RemoveContainer" containerID="20c18c028d05a29a3d45f681554e95cf34d5bbe9071132ec3225acad81c13c2c"
Nov 03 21:44:51.653053 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.653035    5053 scope.go:122] "RemoveContainer" containerID="16be4c649a9cbd5baa803ba185f0dcc51e312bf4e55767a02b20d64a20d72882"
Nov 03 21:44:51.663620 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.663597    5053 scope.go:122] "RemoveContainer" containerID="20c18c028d05a29a3d45f681554e95cf34d5bbe9071132ec3225acad81c13c2c"
Nov 03 21:44:51.664031 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:44:51.664000    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20c18c028d05a29a3d45f681554e95cf34d5bbe9071132ec3225acad81c13c2c\": container with ID starting with 20c18c028d05a29a3d45f681554e95cf34d5bbe9071132ec3225acad81c13c2c not found: ID does not exist" containerID="20c18c028d05a29a3d45f681554e95cf34d5bbe9071132ec3225acad81c13c2c"
Nov 03 21:44:51.664161 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.664041    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c18c028d05a29a3d45f681554e95cf34d5bbe9071132ec3225acad81c13c2c"} err="failed to get container status \"20c18c028d05a29a3d45f681554e95cf34d5bbe9071132ec3225acad81c13c2c\": rpc error: code = NotFound desc = could not find container \"20c18c028d05a29a3d45f681554e95cf34d5bbe9071132ec3225acad81c13c2c\": container with ID starting with 20c18c028d05a29a3d45f681554e95cf34d5bbe9071132ec3225acad81c13c2c not found: ID does not exist"
Nov 03 21:44:51.664161 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.664068    5053 scope.go:122] "RemoveContainer" containerID="16be4c649a9cbd5baa803ba185f0dcc51e312bf4e55767a02b20d64a20d72882"
Nov 03 21:44:51.664333 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:44:51.664315    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16be4c649a9cbd5baa803ba185f0dcc51e312bf4e55767a02b20d64a20d72882\": container with ID starting with 16be4c649a9cbd5baa803ba185f0dcc51e312bf4e55767a02b20d64a20d72882 not found: ID does not exist" containerID="16be4c649a9cbd5baa803ba185f0dcc51e312bf4e55767a02b20d64a20d72882"
Nov 03 21:44:51.664379 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.664340    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16be4c649a9cbd5baa803ba185f0dcc51e312bf4e55767a02b20d64a20d72882"} err="failed to get container status \"16be4c649a9cbd5baa803ba185f0dcc51e312bf4e55767a02b20d64a20d72882\": rpc error: code = NotFound desc = could not find container \"16be4c649a9cbd5baa803ba185f0dcc51e312bf4e55767a02b20d64a20d72882\": container with ID starting with 16be4c649a9cbd5baa803ba185f0dcc51e312bf4e55767a02b20d64a20d72882 not found: ID does not exist"
Nov 03 21:44:51.803247 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.803207    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"]
Nov 03 21:44:51.853278 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:51.853230    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-fff189-openshift-default-8557fcf55c-txjqq"]
Nov 03 21:44:53.669393 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:53.669351    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83a53cc7-96e4-49db-a8ac-2d2de9e84188" path="/var/lib/kubelet/pods/83a53cc7-96e4-49db-a8ac-2d2de9e84188/volumes"
Nov 03 21:44:57.295551 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:57.295517    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn"]
Nov 03 21:44:57.405108 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:44:57.405071    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5"]
Nov 03 21:45:00.180263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.179849    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm"]
Nov 03 21:45:00.184303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.181745    5053 cpu_manager.go:414] "containerMap: removing container" podUID="c61761e5-ce80-491c-b6f0-ed22df43ad19" containerName="revoke-keys"
Nov 03 21:45:00.184303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.181776    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c61761e5-ce80-491c-b6f0-ed22df43ad19" containerName="revoke-keys"
Nov 03 21:45:00.184303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.181814    5053 cpu_manager.go:414] "containerMap: removing container" podUID="83a53cc7-96e4-49db-a8ac-2d2de9e84188" containerName="istio-proxy"
Nov 03 21:45:00.184303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.181822    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a53cc7-96e4-49db-a8ac-2d2de9e84188" containerName="istio-proxy"
Nov 03 21:45:00.184303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.181838    5053 cpu_manager.go:414] "containerMap: removing container" podUID="d64bdf87-f4f6-4015-9d08-f7d4feeb18fc" containerName="revoke-keys"
Nov 03 21:45:00.184303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.181846    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d64bdf87-f4f6-4015-9d08-f7d4feeb18fc" containerName="revoke-keys"
Nov 03 21:45:00.184303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.181882    5053 cpu_manager.go:414] "containerMap: removing container" podUID="83a53cc7-96e4-49db-a8ac-2d2de9e84188" containerName="istio-proxy"
Nov 03 21:45:00.184303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.181890    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a53cc7-96e4-49db-a8ac-2d2de9e84188" containerName="istio-proxy"
Nov 03 21:45:00.184303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.182050    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a53cc7-96e4-49db-a8ac-2d2de9e84188" containerName="istio-proxy"
Nov 03 21:45:00.184303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.182069    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="c61761e5-ce80-491c-b6f0-ed22df43ad19" containerName="revoke-keys"
Nov 03 21:45:00.184303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.182083    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a53cc7-96e4-49db-a8ac-2d2de9e84188" containerName="istio-proxy"
Nov 03 21:45:00.184303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.182091    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="d64bdf87-f4f6-4015-9d08-f7d4feeb18fc" containerName="revoke-keys"
Nov 03 21:45:00.188079 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.187959    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq"]
Nov 03 21:45:00.188225 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.188091    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm"
Nov 03 21:45:00.194132 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.194110    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6"]
Nov 03 21:45:00.194270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.194222    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq"
Nov 03 21:45:00.198662 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.198642    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6"
Nov 03 21:45:00.239763 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.239719    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:45:00.261174 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.261139    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd"]
Nov 03 21:45:00.279595 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.279568    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f"]
Nov 03 21:45:00.398541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.332381    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd"
Nov 03 21:45:00.398541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.351197    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpkl7\" (UniqueName: \"kubernetes.io/projected/932934f1-361d-4768-b972-0a648503c085-kube-api-access-lpkl7\") pod \"maas-api-key-cleanup-29895705-kn9w6\" (UID: \"932934f1-361d-4768-b972-0a648503c085\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6"
Nov 03 21:45:00.398541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.351371    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65569e2b-3978-464d-bec3-ecd8ad99836d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm\" (UID: \"65569e2b-3978-464d-bec3-ecd8ad99836d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm"
Nov 03 21:45:00.398541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.351442    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsm2m\" (UniqueName: \"kubernetes.io/projected/b0cecdca-7f3c-4661-baac-6e3aefda5ce0-kube-api-access-dsm2m\") pod \"maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq\" (UID: \"b0cecdca-7f3c-4661-baac-6e3aefda5ce0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq"
Nov 03 21:45:00.398541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.351537    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932934f1-361d-4768-b972-0a648503c085-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895705-kn9w6\" (UID: \"932934f1-361d-4768-b972-0a648503c085\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6"
Nov 03 21:45:00.398541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.351604    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmbgl\" (UniqueName: \"kubernetes.io/projected/65569e2b-3978-464d-bec3-ecd8ad99836d-kube-api-access-cmbgl\") pod \"maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm\" (UID: \"65569e2b-3978-464d-bec3-ecd8ad99836d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm"
Nov 03 21:45:00.398541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.351661    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cecdca-7f3c-4661-baac-6e3aefda5ce0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq\" (UID: \"b0cecdca-7f3c-4661-baac-6e3aefda5ce0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq"
Nov 03 21:45:00.476984 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.476943    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpkl7\" (UniqueName: \"kubernetes.io/projected/932934f1-361d-4768-b972-0a648503c085-kube-api-access-lpkl7\") pod \"maas-api-key-cleanup-29895705-kn9w6\" (UID: \"932934f1-361d-4768-b972-0a648503c085\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6"
Nov 03 21:45:00.477370 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.477343    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65569e2b-3978-464d-bec3-ecd8ad99836d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm\" (UID: \"65569e2b-3978-464d-bec3-ecd8ad99836d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm"
Nov 03 21:45:00.477551 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.477532    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsm2m\" (UniqueName: \"kubernetes.io/projected/b0cecdca-7f3c-4661-baac-6e3aefda5ce0-kube-api-access-dsm2m\") pod \"maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq\" (UID: \"b0cecdca-7f3c-4661-baac-6e3aefda5ce0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq"
Nov 03 21:45:00.477942 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.477694    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xg8v\" (UniqueName: \"kubernetes.io/projected/d110d78c-27ee-4e76-9758-d92d9087d3a7-kube-api-access-5xg8v\") pod \"maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd\" (UID: \"d110d78c-27ee-4e76-9758-d92d9087d3a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd"
Nov 03 21:45:00.477942 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.477754    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d110d78c-27ee-4e76-9758-d92d9087d3a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd\" (UID: \"d110d78c-27ee-4e76-9758-d92d9087d3a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd"
Nov 03 21:45:00.477942 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.477828    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932934f1-361d-4768-b972-0a648503c085-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895705-kn9w6\" (UID: \"932934f1-361d-4768-b972-0a648503c085\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6"
Nov 03 21:45:00.477942 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.477901    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmbgl\" (UniqueName: \"kubernetes.io/projected/65569e2b-3978-464d-bec3-ecd8ad99836d-kube-api-access-cmbgl\") pod \"maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm\" (UID: \"65569e2b-3978-464d-bec3-ecd8ad99836d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm"
Nov 03 21:45:00.478205 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.478017    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cecdca-7f3c-4661-baac-6e3aefda5ce0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq\" (UID: \"b0cecdca-7f3c-4661-baac-6e3aefda5ce0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq"
Nov 03 21:45:00.539661 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.478892    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cecdca-7f3c-4661-baac-6e3aefda5ce0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq\" (UID: \"b0cecdca-7f3c-4661-baac-6e3aefda5ce0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq"
Nov 03 21:45:00.539661 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.479022    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932934f1-361d-4768-b972-0a648503c085-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895705-kn9w6\" (UID: \"932934f1-361d-4768-b972-0a648503c085\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6"
Nov 03 21:45:00.539661 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.479897    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65569e2b-3978-464d-bec3-ecd8ad99836d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm\" (UID: \"65569e2b-3978-464d-bec3-ecd8ad99836d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm"
Nov 03 21:45:00.539661 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.494472    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm"]
Nov 03 21:45:00.539661 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.494590    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f"
Nov 03 21:45:00.591509 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.591421    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd"]
Nov 03 21:45:00.591509 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.591503    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq"]
Nov 03 21:45:00.591804 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.591532    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6"]
Nov 03 21:45:00.594505 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.594480    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmbgl\" (UniqueName: \"kubernetes.io/projected/65569e2b-3978-464d-bec3-ecd8ad99836d-kube-api-access-cmbgl\") pod \"maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm\" (UID: \"65569e2b-3978-464d-bec3-ecd8ad99836d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm"
Nov 03 21:45:00.595168 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.595122    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4nhx\" (UniqueName: \"kubernetes.io/projected/287cc853-6b9b-42c7-800e-6c0d8eca5d9e-kube-api-access-r4nhx\") pod \"maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f\" (UID: \"287cc853-6b9b-42c7-800e-6c0d8eca5d9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f"
Nov 03 21:45:00.595664 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.595632    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xg8v\" (UniqueName: \"kubernetes.io/projected/d110d78c-27ee-4e76-9758-d92d9087d3a7-kube-api-access-5xg8v\") pod \"maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd\" (UID: \"d110d78c-27ee-4e76-9758-d92d9087d3a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd"
Nov 03 21:45:00.595750 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.595684    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/287cc853-6b9b-42c7-800e-6c0d8eca5d9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f\" (UID: \"287cc853-6b9b-42c7-800e-6c0d8eca5d9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f"
Nov 03 21:45:00.595750 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.595730    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d110d78c-27ee-4e76-9758-d92d9087d3a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd\" (UID: \"d110d78c-27ee-4e76-9758-d92d9087d3a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd"
Nov 03 21:45:00.596884 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.596861    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d110d78c-27ee-4e76-9758-d92d9087d3a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd\" (UID: \"d110d78c-27ee-4e76-9758-d92d9087d3a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd"
Nov 03 21:45:00.642260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.641999    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f"]
Nov 03 21:45:00.677010 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.671907    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpkl7\" (UniqueName: \"kubernetes.io/projected/932934f1-361d-4768-b972-0a648503c085-kube-api-access-lpkl7\") pod \"maas-api-key-cleanup-29895705-kn9w6\" (UID: \"932934f1-361d-4768-b972-0a648503c085\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6"
Nov 03 21:45:00.681181 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.681148    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsm2m\" (UniqueName: \"kubernetes.io/projected/b0cecdca-7f3c-4661-baac-6e3aefda5ce0-kube-api-access-dsm2m\") pod \"maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq\" (UID: \"b0cecdca-7f3c-4661-baac-6e3aefda5ce0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq"
Nov 03 21:45:00.685169 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.685127    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xg8v\" (UniqueName: \"kubernetes.io/projected/d110d78c-27ee-4e76-9758-d92d9087d3a7-kube-api-access-5xg8v\") pod \"maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd\" (UID: \"d110d78c-27ee-4e76-9758-d92d9087d3a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd"
Nov 03 21:45:00.701692 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.696587    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4nhx\" (UniqueName: \"kubernetes.io/projected/287cc853-6b9b-42c7-800e-6c0d8eca5d9e-kube-api-access-r4nhx\") pod \"maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f\" (UID: \"287cc853-6b9b-42c7-800e-6c0d8eca5d9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f"
Nov 03 21:45:00.701692 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.696630    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/287cc853-6b9b-42c7-800e-6c0d8eca5d9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f\" (UID: \"287cc853-6b9b-42c7-800e-6c0d8eca5d9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f"
Nov 03 21:45:00.701692 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.698688    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/287cc853-6b9b-42c7-800e-6c0d8eca5d9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f\" (UID: \"287cc853-6b9b-42c7-800e-6c0d8eca5d9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f"
Nov 03 21:45:00.764224 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.764148    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4nhx\" (UniqueName: \"kubernetes.io/projected/287cc853-6b9b-42c7-800e-6c0d8eca5d9e-kube-api-access-r4nhx\") pod \"maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f\" (UID: \"287cc853-6b9b-42c7-800e-6c0d8eca5d9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f"
Nov 03 21:45:00.841313 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.841270    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd"
Nov 03 21:45:00.876516 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.876485    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm"
Nov 03 21:45:00.884310 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.884285    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq"
Nov 03 21:45:00.930192 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:00.929855    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6"
Nov 03 21:45:01.018871 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:01.018794    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f"
Nov 03 21:45:01.226836 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:01.226804    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd"]
Nov 03 21:45:01.228250 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:45:01.228211    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd110d78c_27ee_4e76_9758_d92d9087d3a7.slice/crio-67de70e727a853592a60ef8d5fdc6be2a822511dce737fcae3537693cc81c461 WatchSource:0}: Error finding container 67de70e727a853592a60ef8d5fdc6be2a822511dce737fcae3537693cc81c461: Status 404 returned error can't find the container with id 67de70e727a853592a60ef8d5fdc6be2a822511dce737fcae3537693cc81c461
Nov 03 21:45:01.281386 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:01.281361    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm"]
Nov 03 21:45:01.281834 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:45:01.281814    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65569e2b_3978_464d_bec3_ecd8ad99836d.slice/crio-bf33607c1348a0808a404ed8766d568bfc7906ede1f49c435ae38480d8de5e01 WatchSource:0}: Error finding container bf33607c1348a0808a404ed8766d568bfc7906ede1f49c435ae38480d8de5e01: Status 404 returned error can't find the container with id bf33607c1348a0808a404ed8766d568bfc7906ede1f49c435ae38480d8de5e01
Nov 03 21:45:01.332745 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:45:01.332717    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0cecdca_7f3c_4661_baac_6e3aefda5ce0.slice/crio-75ebbe92be9a1044bf6e3c347377366a626c1874238a61f6c5fe7c018751f574 WatchSource:0}: Error finding container 75ebbe92be9a1044bf6e3c347377366a626c1874238a61f6c5fe7c018751f574: Status 404 returned error can't find the container with id 75ebbe92be9a1044bf6e3c347377366a626c1874238a61f6c5fe7c018751f574
Nov 03 21:45:01.333829 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:01.333803    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq"]
Nov 03 21:45:01.342062 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:01.342039    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6"]
Nov 03 21:45:01.420426 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:01.420391    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f"]
Nov 03 21:45:01.505874 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:45:01.505833    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod287cc853_6b9b_42c7_800e_6c0d8eca5d9e.slice/crio-0a833e639018e8be05e51a0e88d94469d46a4d7ee72d51769252acd1a8dc8a4d WatchSource:0}: Error finding container 0a833e639018e8be05e51a0e88d94469d46a4d7ee72d51769252acd1a8dc8a4d: Status 404 returned error can't find the container with id 0a833e639018e8be05e51a0e88d94469d46a4d7ee72d51769252acd1a8dc8a4d
Nov 03 21:45:01.782327 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:01.782259    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f" event={"ID":"287cc853-6b9b-42c7-800e-6c0d8eca5d9e","Type":"ContainerStarted","Data":"0a833e639018e8be05e51a0e88d94469d46a4d7ee72d51769252acd1a8dc8a4d"}
Nov 03 21:45:01.784222 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:01.784189    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd" event={"ID":"d110d78c-27ee-4e76-9758-d92d9087d3a7","Type":"ContainerStarted","Data":"df9607a564c8ea829e614249eb42ae7f498056538f3f3b2d5e415d3fa68ba595"}
Nov 03 21:45:01.784422 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:01.784266    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd" event={"ID":"d110d78c-27ee-4e76-9758-d92d9087d3a7","Type":"ContainerStarted","Data":"67de70e727a853592a60ef8d5fdc6be2a822511dce737fcae3537693cc81c461"}
Nov 03 21:45:01.786014 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:01.785991    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm" event={"ID":"65569e2b-3978-464d-bec3-ecd8ad99836d","Type":"ContainerStarted","Data":"1948a29b06d81df1a5b3702a0fa5dca82309c69085a7cdda74157d6e1e6a9803"}
Nov 03 21:45:01.786130 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:01.786038    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm" event={"ID":"65569e2b-3978-464d-bec3-ecd8ad99836d","Type":"ContainerStarted","Data":"bf33607c1348a0808a404ed8766d568bfc7906ede1f49c435ae38480d8de5e01"}
Nov 03 21:45:01.787292 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:01.787258    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6" event={"ID":"932934f1-361d-4768-b972-0a648503c085","Type":"ContainerStarted","Data":"5e017b89f65f985d758553b24c5dad1a0da3a66cc43ffae4a969d230634c54a9"}
Nov 03 21:45:01.788684 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:01.788661    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq" event={"ID":"b0cecdca-7f3c-4661-baac-6e3aefda5ce0","Type":"ContainerStarted","Data":"75ebbe92be9a1044bf6e3c347377366a626c1874238a61f6c5fe7c018751f574"}
Nov 03 21:45:01.900261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:01.900178    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd" podStartSLOduration=1.900162966 podStartE2EDuration="1.900162966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:01.896764311 +0000 UTC m=+1816.390686230" watchObservedRunningTime="2026-11-03 21:45:01.900162966 +0000 UTC m=+1816.394084840"
Nov 03 21:45:02.010936 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.010889    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm" podStartSLOduration=2.010876216 podStartE2EDuration="2.010876216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:02.008715617 +0000 UTC m=+1816.502637492" watchObservedRunningTime="2026-11-03 21:45:02.010876216 +0000 UTC m=+1816.504798114"
Nov 03 21:45:02.296228 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.296183    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn" podUID="c02684d1-aaec-4345-9ce7-4d910ed4fbe9" containerName="payload-processing" containerID="cri-o://643b4b1c12b849298908c1fe70121643a007aa705e8b82c1f00f934b9126de1b" gracePeriod=25
Nov 03 21:45:02.406102 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.406035    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5" podUID="6a7b45d4-526f-4568-8036-1a614c502fe5" containerName="payload-pre-processing" containerID="cri-o://600abcac298be4df0f73f08519a4a25dc3da7ec60e2644dafac732388642373c" gracePeriod=25
Nov 03 21:45:02.589480 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.589451    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn"
Nov 03 21:45:02.627831 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.627797    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c02684d1-aaec-4345-9ce7-4d910ed4fbe9-kube-api-access-g7xn7\" (UniqueName: \"kubernetes.io/projected/c02684d1-aaec-4345-9ce7-4d910ed4fbe9-kube-api-access-g7xn7\") pod \"c02684d1-aaec-4345-9ce7-4d910ed4fbe9\" (UID: \"c02684d1-aaec-4345-9ce7-4d910ed4fbe9\") "
Nov 03 21:45:02.628593 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.628001    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c02684d1-aaec-4345-9ce7-4d910ed4fbe9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c02684d1-aaec-4345-9ce7-4d910ed4fbe9-plugins-config-volume\") pod \"c02684d1-aaec-4345-9ce7-4d910ed4fbe9\" (UID: \"c02684d1-aaec-4345-9ce7-4d910ed4fbe9\") "
Nov 03 21:45:02.628774 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.628748    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c02684d1-aaec-4345-9ce7-4d910ed4fbe9-plugins-config-volume" pod "c02684d1-aaec-4345-9ce7-4d910ed4fbe9" (UID: "c02684d1-aaec-4345-9ce7-4d910ed4fbe9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:02.632023 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.631943    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c02684d1-aaec-4345-9ce7-4d910ed4fbe9-kube-api-access-g7xn7" pod "c02684d1-aaec-4345-9ce7-4d910ed4fbe9" (UID: "c02684d1-aaec-4345-9ce7-4d910ed4fbe9"). InnerVolumeSpecName "kube-api-access-g7xn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:02.721140 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.721107    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5"
Nov 03 21:45:02.728945 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.728916    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a7b45d4-526f-4568-8036-1a614c502fe5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a7b45d4-526f-4568-8036-1a614c502fe5-plugins-config-volume\") pod \"6a7b45d4-526f-4568-8036-1a614c502fe5\" (UID: \"6a7b45d4-526f-4568-8036-1a614c502fe5\") "
Nov 03 21:45:02.729146 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.729093    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a7b45d4-526f-4568-8036-1a614c502fe5-kube-api-access-5558z\" (UniqueName: \"kubernetes.io/projected/6a7b45d4-526f-4568-8036-1a614c502fe5-kube-api-access-5558z\") pod \"6a7b45d4-526f-4568-8036-1a614c502fe5\" (UID: \"6a7b45d4-526f-4568-8036-1a614c502fe5\") "
Nov 03 21:45:02.729417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.729388    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7b45d4-526f-4568-8036-1a614c502fe5-plugins-config-volume" pod "6a7b45d4-526f-4568-8036-1a614c502fe5" (UID: "6a7b45d4-526f-4568-8036-1a614c502fe5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:02.729514 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.729441    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7xn7\" (UniqueName: \"kubernetes.io/projected/c02684d1-aaec-4345-9ce7-4d910ed4fbe9-kube-api-access-g7xn7\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:02.729514 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.729464    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c02684d1-aaec-4345-9ce7-4d910ed4fbe9-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:02.731976 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.731948    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7b45d4-526f-4568-8036-1a614c502fe5-kube-api-access-5558z" pod "6a7b45d4-526f-4568-8036-1a614c502fe5" (UID: "6a7b45d4-526f-4568-8036-1a614c502fe5"). InnerVolumeSpecName "kube-api-access-5558z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:02.797113 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.797081    5053 generic.go:356] "Generic (PLEG): container finished" podID="6a7b45d4-526f-4568-8036-1a614c502fe5" containerID="600abcac298be4df0f73f08519a4a25dc3da7ec60e2644dafac732388642373c" exitCode=0
Nov 03 21:45:02.797364 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.797131    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5"
Nov 03 21:45:02.797364 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.797172    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5" event={"ID":"6a7b45d4-526f-4568-8036-1a614c502fe5","Type":"ContainerDied","Data":"600abcac298be4df0f73f08519a4a25dc3da7ec60e2644dafac732388642373c"}
Nov 03 21:45:02.797364 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.797224    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5" event={"ID":"6a7b45d4-526f-4568-8036-1a614c502fe5","Type":"ContainerDied","Data":"e8041d572033bf1d072396b49b9198c0877efc98ba2aefec582a34a808005e3d"}
Nov 03 21:45:02.797364 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.797264    5053 scope.go:122] "RemoveContainer" containerID="600abcac298be4df0f73f08519a4a25dc3da7ec60e2644dafac732388642373c"
Nov 03 21:45:02.799135 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.799111    5053 generic.go:356] "Generic (PLEG): container finished" podID="65569e2b-3978-464d-bec3-ecd8ad99836d" containerID="1948a29b06d81df1a5b3702a0fa5dca82309c69085a7cdda74157d6e1e6a9803" exitCode=60
Nov 03 21:45:02.799308 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.799181    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm" event={"ID":"65569e2b-3978-464d-bec3-ecd8ad99836d","Type":"ContainerDied","Data":"1948a29b06d81df1a5b3702a0fa5dca82309c69085a7cdda74157d6e1e6a9803"}
Nov 03 21:45:02.799575 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.799558    5053 scope.go:122] "RemoveContainer" containerID="1948a29b06d81df1a5b3702a0fa5dca82309c69085a7cdda74157d6e1e6a9803"
Nov 03 21:45:02.801296 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.801170    5053 generic.go:356] "Generic (PLEG): container finished" podID="932934f1-361d-4768-b972-0a648503c085" containerID="997a285d714e6c9db0f3573b9f225228ed969f6ed98916f69f2ec68028de79c9" exitCode=60
Nov 03 21:45:02.801383 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.801324    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6" event={"ID":"932934f1-361d-4768-b972-0a648503c085","Type":"ContainerDied","Data":"997a285d714e6c9db0f3573b9f225228ed969f6ed98916f69f2ec68028de79c9"}
Nov 03 21:45:02.801552 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.801537    5053 scope.go:122] "RemoveContainer" containerID="997a285d714e6c9db0f3573b9f225228ed969f6ed98916f69f2ec68028de79c9"
Nov 03 21:45:02.803737 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.803374    5053 generic.go:356] "Generic (PLEG): container finished" podID="b0cecdca-7f3c-4661-baac-6e3aefda5ce0" containerID="a6f54cc201feea6177a9c7fa67a48502bf21e0725479e0fcf04b1e4fdc0492fb" exitCode=60
Nov 03 21:45:02.803737 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.803545    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq" event={"ID":"b0cecdca-7f3c-4661-baac-6e3aefda5ce0","Type":"ContainerDied","Data":"a6f54cc201feea6177a9c7fa67a48502bf21e0725479e0fcf04b1e4fdc0492fb"}
Nov 03 21:45:02.803737 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.803715    5053 scope.go:122] "RemoveContainer" containerID="a6f54cc201feea6177a9c7fa67a48502bf21e0725479e0fcf04b1e4fdc0492fb"
Nov 03 21:45:02.805366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.805343    5053 generic.go:356] "Generic (PLEG): container finished" podID="287cc853-6b9b-42c7-800e-6c0d8eca5d9e" containerID="00ae103688c399275d6f57c72de628b5560562c2ce9e031341074cac67b3e62d" exitCode=60
Nov 03 21:45:02.805474 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.805456    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f" event={"ID":"287cc853-6b9b-42c7-800e-6c0d8eca5d9e","Type":"ContainerDied","Data":"00ae103688c399275d6f57c72de628b5560562c2ce9e031341074cac67b3e62d"}
Nov 03 21:45:02.805601 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.805585    5053 scope.go:122] "RemoveContainer" containerID="00ae103688c399275d6f57c72de628b5560562c2ce9e031341074cac67b3e62d"
Nov 03 21:45:02.806979 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.806958    5053 generic.go:356] "Generic (PLEG): container finished" podID="c02684d1-aaec-4345-9ce7-4d910ed4fbe9" containerID="643b4b1c12b849298908c1fe70121643a007aa705e8b82c1f00f934b9126de1b" exitCode=0
Nov 03 21:45:02.807060 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.807003    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn"
Nov 03 21:45:02.807060 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.807018    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn" event={"ID":"c02684d1-aaec-4345-9ce7-4d910ed4fbe9","Type":"ContainerDied","Data":"643b4b1c12b849298908c1fe70121643a007aa705e8b82c1f00f934b9126de1b"}
Nov 03 21:45:02.807060 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.807055    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn" event={"ID":"c02684d1-aaec-4345-9ce7-4d910ed4fbe9","Type":"ContainerDied","Data":"8dce1aeaf3032b086d2d440828517f0bc650adf8147cc9d00c3e156f9ac2808c"}
Nov 03 21:45:02.808854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.808835    5053 generic.go:356] "Generic (PLEG): container finished" podID="d110d78c-27ee-4e76-9758-d92d9087d3a7" containerID="df9607a564c8ea829e614249eb42ae7f498056538f3f3b2d5e415d3fa68ba595" exitCode=60
Nov 03 21:45:02.808966 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.808877    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd" event={"ID":"d110d78c-27ee-4e76-9758-d92d9087d3a7","Type":"ContainerDied","Data":"df9607a564c8ea829e614249eb42ae7f498056538f3f3b2d5e415d3fa68ba595"}
Nov 03 21:45:02.809174 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.809157    5053 scope.go:122] "RemoveContainer" containerID="df9607a564c8ea829e614249eb42ae7f498056538f3f3b2d5e415d3fa68ba595"
Nov 03 21:45:02.821998 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.821979    5053 scope.go:122] "RemoveContainer" containerID="600abcac298be4df0f73f08519a4a25dc3da7ec60e2644dafac732388642373c"
Nov 03 21:45:02.822381 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:02.822357    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"600abcac298be4df0f73f08519a4a25dc3da7ec60e2644dafac732388642373c\": container with ID starting with 600abcac298be4df0f73f08519a4a25dc3da7ec60e2644dafac732388642373c not found: ID does not exist" containerID="600abcac298be4df0f73f08519a4a25dc3da7ec60e2644dafac732388642373c"
Nov 03 21:45:02.822474 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.822391    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"600abcac298be4df0f73f08519a4a25dc3da7ec60e2644dafac732388642373c"} err="failed to get container status \"600abcac298be4df0f73f08519a4a25dc3da7ec60e2644dafac732388642373c\": rpc error: code = NotFound desc = could not find container \"600abcac298be4df0f73f08519a4a25dc3da7ec60e2644dafac732388642373c\": container with ID starting with 600abcac298be4df0f73f08519a4a25dc3da7ec60e2644dafac732388642373c not found: ID does not exist"
Nov 03 21:45:02.822474 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.822416    5053 scope.go:122] "RemoveContainer" containerID="643b4b1c12b849298908c1fe70121643a007aa705e8b82c1f00f934b9126de1b"
Nov 03 21:45:02.831081 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.831057    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5558z\" (UniqueName: \"kubernetes.io/projected/6a7b45d4-526f-4568-8036-1a614c502fe5-kube-api-access-5558z\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:02.831081 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.831082    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a7b45d4-526f-4568-8036-1a614c502fe5-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:02.851227 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.851203    5053 scope.go:122] "RemoveContainer" containerID="643b4b1c12b849298908c1fe70121643a007aa705e8b82c1f00f934b9126de1b"
Nov 03 21:45:02.851607 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:02.851587    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"643b4b1c12b849298908c1fe70121643a007aa705e8b82c1f00f934b9126de1b\": container with ID starting with 643b4b1c12b849298908c1fe70121643a007aa705e8b82c1f00f934b9126de1b not found: ID does not exist" containerID="643b4b1c12b849298908c1fe70121643a007aa705e8b82c1f00f934b9126de1b"
Nov 03 21:45:02.851674 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.851617    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"643b4b1c12b849298908c1fe70121643a007aa705e8b82c1f00f934b9126de1b"} err="failed to get container status \"643b4b1c12b849298908c1fe70121643a007aa705e8b82c1f00f934b9126de1b\": rpc error: code = NotFound desc = could not find container \"643b4b1c12b849298908c1fe70121643a007aa705e8b82c1f00f934b9126de1b\": container with ID starting with 643b4b1c12b849298908c1fe70121643a007aa705e8b82c1f00f934b9126de1b not found: ID does not exist"
Nov 03 21:45:02.948946 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.948912    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn"]
Nov 03 21:45:02.953780 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:02.953752    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-89ddad-67d59d6676-6kzsn"]
Nov 03 21:45:03.064599 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:03.064563    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5"]
Nov 03 21:45:03.071628 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:03.071598    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-89ddad-799d44d47-zzmg5"]
Nov 03 21:45:03.668658 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:03.668579    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a7b45d4-526f-4568-8036-1a614c502fe5" path="/var/lib/kubelet/pods/6a7b45d4-526f-4568-8036-1a614c502fe5/volumes"
Nov 03 21:45:03.669088 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:03.668993    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c02684d1-aaec-4345-9ce7-4d910ed4fbe9" path="/var/lib/kubelet/pods/c02684d1-aaec-4345-9ce7-4d910ed4fbe9/volumes"
Nov 03 21:45:03.823050 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:03.823020    5053 generic.go:356] "Generic (PLEG): container finished" podID="65569e2b-3978-464d-bec3-ecd8ad99836d" containerID="ab4cdf84ab4dbd30fc1287536140775c9c2d8092121aebe673cecc02a4b763f8" exitCode=60
Nov 03 21:45:03.823266 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:03.823094    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm" event={"ID":"65569e2b-3978-464d-bec3-ecd8ad99836d","Type":"ContainerDied","Data":"ab4cdf84ab4dbd30fc1287536140775c9c2d8092121aebe673cecc02a4b763f8"}
Nov 03 21:45:03.823266 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:03.823161    5053 scope.go:122] "RemoveContainer" containerID="1948a29b06d81df1a5b3702a0fa5dca82309c69085a7cdda74157d6e1e6a9803"
Nov 03 21:45:03.823516 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:03.823495    5053 scope.go:122] "RemoveContainer" containerID="ab4cdf84ab4dbd30fc1287536140775c9c2d8092121aebe673cecc02a4b763f8"
Nov 03 21:45:03.823900 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:03.823870    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm_odh-ai-gateway-infra(65569e2b-3978-464d-bec3-ecd8ad99836d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm" podUID="65569e2b-3978-464d-bec3-ecd8ad99836d"
Nov 03 21:45:03.825329 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:03.825305    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6" event={"ID":"932934f1-361d-4768-b972-0a648503c085","Type":"ContainerStarted","Data":"66d3109ef5e3280066ab931102036d57abfe6b72729d29c4c4d843bd74121f70"}
Nov 03 21:45:03.825565 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:03.825552    5053 scope.go:122] "RemoveContainer" containerID="66d3109ef5e3280066ab931102036d57abfe6b72729d29c4c4d843bd74121f70"
Nov 03 21:45:03.825855 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:03.825832    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895705-kn9w6_odh-ai-gateway-infra(932934f1-361d-4768-b972-0a648503c085)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6" podUID="932934f1-361d-4768-b972-0a648503c085"
Nov 03 21:45:03.827527 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:03.827505    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq" event={"ID":"b0cecdca-7f3c-4661-baac-6e3aefda5ce0","Type":"ContainerStarted","Data":"62fe05119bcee49b0b9698df62ee8fc9a921965af40153dd2840cb6e33dbcaa4"}
Nov 03 21:45:03.829775 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:03.829750    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f" event={"ID":"287cc853-6b9b-42c7-800e-6c0d8eca5d9e","Type":"ContainerStarted","Data":"11c91feb834d3138b1a19d67c41d821a388577ecb2ffb5d02ad30a6f9854c054"}
Nov 03 21:45:03.833485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:03.833457    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd" event={"ID":"d110d78c-27ee-4e76-9758-d92d9087d3a7","Type":"ContainerStarted","Data":"4e0b614daeb6aa3cb282cea1672c4aa0dd6475ab1968e64333728e9ba14675b2"}
Nov 03 21:45:03.833721 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:03.833696    5053 scope.go:122] "RemoveContainer" containerID="4e0b614daeb6aa3cb282cea1672c4aa0dd6475ab1968e64333728e9ba14675b2"
Nov 03 21:45:03.833967 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:03.833944    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd_odh-ai-gateway-infra(d110d78c-27ee-4e76-9758-d92d9087d3a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd" podUID="d110d78c-27ee-4e76-9758-d92d9087d3a7"
Nov 03 21:45:04.054056 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:04.054002    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f" podStartSLOduration=4.053987015 podStartE2EDuration="4.053987015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:04.050153495 +0000 UTC m=+1818.544075369" watchObservedRunningTime="2026-11-03 21:45:04.053987015 +0000 UTC m=+1818.547908889"
Nov 03 21:45:04.414690 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:04.414609    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq" podStartSLOduration=4.414573906 podStartE2EDuration="4.414573906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:04.297720488 +0000 UTC m=+1818.791642361" watchObservedRunningTime="2026-11-03 21:45:04.414573906 +0000 UTC m=+1818.908495781"
Nov 03 21:45:04.857620 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:04.857567    5053 generic.go:356] "Generic (PLEG): container finished" podID="287cc853-6b9b-42c7-800e-6c0d8eca5d9e" containerID="11c91feb834d3138b1a19d67c41d821a388577ecb2ffb5d02ad30a6f9854c054" exitCode=60
Nov 03 21:45:04.858108 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:04.857722    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f" event={"ID":"287cc853-6b9b-42c7-800e-6c0d8eca5d9e","Type":"ContainerDied","Data":"11c91feb834d3138b1a19d67c41d821a388577ecb2ffb5d02ad30a6f9854c054"}
Nov 03 21:45:04.858108 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:04.857803    5053 scope.go:122] "RemoveContainer" containerID="00ae103688c399275d6f57c72de628b5560562c2ce9e031341074cac67b3e62d"
Nov 03 21:45:04.858849 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:04.858822    5053 scope.go:122] "RemoveContainer" containerID="11c91feb834d3138b1a19d67c41d821a388577ecb2ffb5d02ad30a6f9854c054"
Nov 03 21:45:04.859344 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:04.859156    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f_odh-ai-gateway-infra(287cc853-6b9b-42c7-800e-6c0d8eca5d9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f" podUID="287cc853-6b9b-42c7-800e-6c0d8eca5d9e"
Nov 03 21:45:04.859978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:04.859959    5053 generic.go:356] "Generic (PLEG): container finished" podID="d110d78c-27ee-4e76-9758-d92d9087d3a7" containerID="4e0b614daeb6aa3cb282cea1672c4aa0dd6475ab1968e64333728e9ba14675b2" exitCode=60
Nov 03 21:45:04.860116 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:04.860097    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd" event={"ID":"d110d78c-27ee-4e76-9758-d92d9087d3a7","Type":"ContainerDied","Data":"4e0b614daeb6aa3cb282cea1672c4aa0dd6475ab1968e64333728e9ba14675b2"}
Nov 03 21:45:04.860417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:04.860395    5053 scope.go:122] "RemoveContainer" containerID="4e0b614daeb6aa3cb282cea1672c4aa0dd6475ab1968e64333728e9ba14675b2"
Nov 03 21:45:04.860671 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:04.860646    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd_odh-ai-gateway-infra(d110d78c-27ee-4e76-9758-d92d9087d3a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd" podUID="d110d78c-27ee-4e76-9758-d92d9087d3a7"
Nov 03 21:45:04.861971 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:04.861955    5053 scope.go:122] "RemoveContainer" containerID="ab4cdf84ab4dbd30fc1287536140775c9c2d8092121aebe673cecc02a4b763f8"
Nov 03 21:45:04.862293 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:04.862268    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm_odh-ai-gateway-infra(65569e2b-3978-464d-bec3-ecd8ad99836d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm" podUID="65569e2b-3978-464d-bec3-ecd8ad99836d"
Nov 03 21:45:04.864597 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:04.864552    5053 generic.go:356] "Generic (PLEG): container finished" podID="932934f1-361d-4768-b972-0a648503c085" containerID="66d3109ef5e3280066ab931102036d57abfe6b72729d29c4c4d843bd74121f70" exitCode=60
Nov 03 21:45:04.864701 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:04.864614    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6" event={"ID":"932934f1-361d-4768-b972-0a648503c085","Type":"ContainerDied","Data":"66d3109ef5e3280066ab931102036d57abfe6b72729d29c4c4d843bd74121f70"}
Nov 03 21:45:04.865022 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:04.864997    5053 scope.go:122] "RemoveContainer" containerID="66d3109ef5e3280066ab931102036d57abfe6b72729d29c4c4d843bd74121f70"
Nov 03 21:45:04.865386 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:04.865360    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895705-kn9w6_odh-ai-gateway-infra(932934f1-361d-4768-b972-0a648503c085)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6" podUID="932934f1-361d-4768-b972-0a648503c085"
Nov 03 21:45:04.866864 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:04.866840    5053 generic.go:356] "Generic (PLEG): container finished" podID="b0cecdca-7f3c-4661-baac-6e3aefda5ce0" containerID="62fe05119bcee49b0b9698df62ee8fc9a921965af40153dd2840cb6e33dbcaa4" exitCode=60
Nov 03 21:45:04.866970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:04.866931    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq" event={"ID":"b0cecdca-7f3c-4661-baac-6e3aefda5ce0","Type":"ContainerDied","Data":"62fe05119bcee49b0b9698df62ee8fc9a921965af40153dd2840cb6e33dbcaa4"}
Nov 03 21:45:04.867416 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:04.867398    5053 scope.go:122] "RemoveContainer" containerID="62fe05119bcee49b0b9698df62ee8fc9a921965af40153dd2840cb6e33dbcaa4"
Nov 03 21:45:04.867635 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:04.867615    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq_odh-ai-gateway-infra(b0cecdca-7f3c-4661-baac-6e3aefda5ce0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq" podUID="b0cecdca-7f3c-4661-baac-6e3aefda5ce0"
Nov 03 21:45:04.872588 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:04.872568    5053 scope.go:122] "RemoveContainer" containerID="df9607a564c8ea829e614249eb42ae7f498056538f3f3b2d5e415d3fa68ba595"
Nov 03 21:45:04.886952 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:04.886932    5053 scope.go:122] "RemoveContainer" containerID="997a285d714e6c9db0f3573b9f225228ed969f6ed98916f69f2ec68028de79c9"
Nov 03 21:45:04.900133 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:04.900109    5053 scope.go:122] "RemoveContainer" containerID="a6f54cc201feea6177a9c7fa67a48502bf21e0725479e0fcf04b1e4fdc0492fb"
Nov 03 21:45:05.878947 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:05.878919    5053 scope.go:122] "RemoveContainer" containerID="4e0b614daeb6aa3cb282cea1672c4aa0dd6475ab1968e64333728e9ba14675b2"
Nov 03 21:45:05.879404 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:05.879258    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd_odh-ai-gateway-infra(d110d78c-27ee-4e76-9758-d92d9087d3a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd" podUID="d110d78c-27ee-4e76-9758-d92d9087d3a7"
Nov 03 21:45:05.880634 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:05.880611    5053 scope.go:122] "RemoveContainer" containerID="66d3109ef5e3280066ab931102036d57abfe6b72729d29c4c4d843bd74121f70"
Nov 03 21:45:05.880918 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:05.880892    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895705-kn9w6_odh-ai-gateway-infra(932934f1-361d-4768-b972-0a648503c085)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6" podUID="932934f1-361d-4768-b972-0a648503c085"
Nov 03 21:45:05.882279 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:05.882228    5053 scope.go:122] "RemoveContainer" containerID="62fe05119bcee49b0b9698df62ee8fc9a921965af40153dd2840cb6e33dbcaa4"
Nov 03 21:45:05.882548 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:05.882529    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq_odh-ai-gateway-infra(b0cecdca-7f3c-4661-baac-6e3aefda5ce0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq" podUID="b0cecdca-7f3c-4661-baac-6e3aefda5ce0"
Nov 03 21:45:05.883781 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:05.883765    5053 scope.go:122] "RemoveContainer" containerID="11c91feb834d3138b1a19d67c41d821a388577ecb2ffb5d02ad30a6f9854c054"
Nov 03 21:45:05.883977 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:05.883961    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f_odh-ai-gateway-infra(287cc853-6b9b-42c7-800e-6c0d8eca5d9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f" podUID="287cc853-6b9b-42c7-800e-6c0d8eca5d9e"
Nov 03 21:45:10.933445 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:10.933400    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:45:14.659381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:14.659354    5053 scope.go:122] "RemoveContainer" containerID="11c91feb834d3138b1a19d67c41d821a388577ecb2ffb5d02ad30a6f9854c054"
Nov 03 21:45:15.668854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:15.668827    5053 scope.go:122] "RemoveContainer" containerID="62fe05119bcee49b0b9698df62ee8fc9a921965af40153dd2840cb6e33dbcaa4"
Nov 03 21:45:15.669272 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:15.668961    5053 scope.go:122] "RemoveContainer" containerID="4e0b614daeb6aa3cb282cea1672c4aa0dd6475ab1968e64333728e9ba14675b2"
Nov 03 21:45:15.957740 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:15.957708    5053 generic.go:356] "Generic (PLEG): container finished" podID="287cc853-6b9b-42c7-800e-6c0d8eca5d9e" containerID="1f150a767b20b35c91cdfd58edbd782b4c46d5e1502608e0ceb0a2b983b4017a" exitCode=60
Nov 03 21:45:15.957909 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:15.957777    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f" event={"ID":"287cc853-6b9b-42c7-800e-6c0d8eca5d9e","Type":"ContainerDied","Data":"1f150a767b20b35c91cdfd58edbd782b4c46d5e1502608e0ceb0a2b983b4017a"}
Nov 03 21:45:15.957909 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:15.957839    5053 scope.go:122] "RemoveContainer" containerID="11c91feb834d3138b1a19d67c41d821a388577ecb2ffb5d02ad30a6f9854c054"
Nov 03 21:45:15.958219 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:15.958194    5053 scope.go:122] "RemoveContainer" containerID="1f150a767b20b35c91cdfd58edbd782b4c46d5e1502608e0ceb0a2b983b4017a"
Nov 03 21:45:15.958573 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:15.958553    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f_odh-ai-gateway-infra(287cc853-6b9b-42c7-800e-6c0d8eca5d9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f" podUID="287cc853-6b9b-42c7-800e-6c0d8eca5d9e"
Nov 03 21:45:16.967079 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:16.967047    5053 generic.go:356] "Generic (PLEG): container finished" podID="b0cecdca-7f3c-4661-baac-6e3aefda5ce0" containerID="6db4e97a5d1690f85d7d252d2a51cf2ac04be47f040f0ffa63d3b79fab47538a" exitCode=60
Nov 03 21:45:16.967576 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:16.967121    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq" event={"ID":"b0cecdca-7f3c-4661-baac-6e3aefda5ce0","Type":"ContainerDied","Data":"6db4e97a5d1690f85d7d252d2a51cf2ac04be47f040f0ffa63d3b79fab47538a"}
Nov 03 21:45:16.967576 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:16.967182    5053 scope.go:122] "RemoveContainer" containerID="62fe05119bcee49b0b9698df62ee8fc9a921965af40153dd2840cb6e33dbcaa4"
Nov 03 21:45:16.967713 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:16.967621    5053 scope.go:122] "RemoveContainer" containerID="6db4e97a5d1690f85d7d252d2a51cf2ac04be47f040f0ffa63d3b79fab47538a"
Nov 03 21:45:16.967985 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:16.967962    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq_odh-ai-gateway-infra(b0cecdca-7f3c-4661-baac-6e3aefda5ce0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq" podUID="b0cecdca-7f3c-4661-baac-6e3aefda5ce0"
Nov 03 21:45:16.970848 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:16.970829    5053 generic.go:356] "Generic (PLEG): container finished" podID="d110d78c-27ee-4e76-9758-d92d9087d3a7" containerID="408ff86d48584d96de5a1effe721c5d4959d7cf6be246ccc389dc03f75af6715" exitCode=60
Nov 03 21:45:16.970946 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:16.970866    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd" event={"ID":"d110d78c-27ee-4e76-9758-d92d9087d3a7","Type":"ContainerDied","Data":"408ff86d48584d96de5a1effe721c5d4959d7cf6be246ccc389dc03f75af6715"}
Nov 03 21:45:16.971194 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:16.971181    5053 scope.go:122] "RemoveContainer" containerID="408ff86d48584d96de5a1effe721c5d4959d7cf6be246ccc389dc03f75af6715"
Nov 03 21:45:16.971468 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:16.971449    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd_odh-ai-gateway-infra(d110d78c-27ee-4e76-9758-d92d9087d3a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd" podUID="d110d78c-27ee-4e76-9758-d92d9087d3a7"
Nov 03 21:45:16.979961 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:16.979943    5053 scope.go:122] "RemoveContainer" containerID="4e0b614daeb6aa3cb282cea1672c4aa0dd6475ab1968e64333728e9ba14675b2"
Nov 03 21:45:17.080631 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:17.080552    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f"]
Nov 03 21:45:17.244128 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:17.244093    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f"
Nov 03 21:45:17.319409 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:17.319379    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/287cc853-6b9b-42c7-800e-6c0d8eca5d9e-kube-api-access-r4nhx\" (UniqueName: \"kubernetes.io/projected/287cc853-6b9b-42c7-800e-6c0d8eca5d9e-kube-api-access-r4nhx\") pod \"287cc853-6b9b-42c7-800e-6c0d8eca5d9e\" (UID: \"287cc853-6b9b-42c7-800e-6c0d8eca5d9e\") "
Nov 03 21:45:17.319566 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:17.319502    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/287cc853-6b9b-42c7-800e-6c0d8eca5d9e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/287cc853-6b9b-42c7-800e-6c0d8eca5d9e-maas-api-service-ca\") pod \"287cc853-6b9b-42c7-800e-6c0d8eca5d9e\" (UID: \"287cc853-6b9b-42c7-800e-6c0d8eca5d9e\") "
Nov 03 21:45:17.319857 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:17.319826    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/287cc853-6b9b-42c7-800e-6c0d8eca5d9e-maas-api-service-ca" pod "287cc853-6b9b-42c7-800e-6c0d8eca5d9e" (UID: "287cc853-6b9b-42c7-800e-6c0d8eca5d9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:17.320112 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:17.320096    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/287cc853-6b9b-42c7-800e-6c0d8eca5d9e-maas-api-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:17.321673 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:17.321650    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/287cc853-6b9b-42c7-800e-6c0d8eca5d9e-kube-api-access-r4nhx" pod "287cc853-6b9b-42c7-800e-6c0d8eca5d9e" (UID: "287cc853-6b9b-42c7-800e-6c0d8eca5d9e"). InnerVolumeSpecName "kube-api-access-r4nhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:17.422189 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:17.422109    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4nhx\" (UniqueName: \"kubernetes.io/projected/287cc853-6b9b-42c7-800e-6c0d8eca5d9e-kube-api-access-r4nhx\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:17.660542 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:17.660510    5053 scope.go:122] "RemoveContainer" containerID="66d3109ef5e3280066ab931102036d57abfe6b72729d29c4c4d843bd74121f70"
Nov 03 21:45:17.661723 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:17.661326    5053 scope.go:122] "RemoveContainer" containerID="ab4cdf84ab4dbd30fc1287536140775c9c2d8092121aebe673cecc02a4b763f8"
Nov 03 21:45:17.981163 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:17.981135    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f"
Nov 03 21:45:17.981653 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:17.981178    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f" event={"ID":"287cc853-6b9b-42c7-800e-6c0d8eca5d9e","Type":"ContainerDied","Data":"0a833e639018e8be05e51a0e88d94469d46a4d7ee72d51769252acd1a8dc8a4d"}
Nov 03 21:45:17.981653 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:17.981261    5053 scope.go:122] "RemoveContainer" containerID="1f150a767b20b35c91cdfd58edbd782b4c46d5e1502608e0ceb0a2b983b4017a"
Nov 03 21:45:18.093079 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.093044    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq"]
Nov 03 21:45:18.145400 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.144798    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f"]
Nov 03 21:45:18.148609 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.148580    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895705-lng5f"]
Nov 03 21:45:18.203010 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.202978    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd"]
Nov 03 21:45:18.263126 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.263097    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq"
Nov 03 21:45:18.331778 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.331743    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0cecdca-7f3c-4661-baac-6e3aefda5ce0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cecdca-7f3c-4661-baac-6e3aefda5ce0-maas-api-service-ca\") pod \"b0cecdca-7f3c-4661-baac-6e3aefda5ce0\" (UID: \"b0cecdca-7f3c-4661-baac-6e3aefda5ce0\") "
Nov 03 21:45:18.331947 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.331898    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0cecdca-7f3c-4661-baac-6e3aefda5ce0-kube-api-access-dsm2m\" (UniqueName: \"kubernetes.io/projected/b0cecdca-7f3c-4661-baac-6e3aefda5ce0-kube-api-access-dsm2m\") pod \"b0cecdca-7f3c-4661-baac-6e3aefda5ce0\" (UID: \"b0cecdca-7f3c-4661-baac-6e3aefda5ce0\") "
Nov 03 21:45:18.332171 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.332146    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0cecdca-7f3c-4661-baac-6e3aefda5ce0-maas-api-service-ca" pod "b0cecdca-7f3c-4661-baac-6e3aefda5ce0" (UID: "b0cecdca-7f3c-4661-baac-6e3aefda5ce0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:18.332404 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.332384    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cecdca-7f3c-4661-baac-6e3aefda5ce0-maas-api-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:18.334223 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.334201    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0cecdca-7f3c-4661-baac-6e3aefda5ce0-kube-api-access-dsm2m" pod "b0cecdca-7f3c-4661-baac-6e3aefda5ce0" (UID: "b0cecdca-7f3c-4661-baac-6e3aefda5ce0"). InnerVolumeSpecName "kube-api-access-dsm2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:18.366499 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.366476    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd"
Nov 03 21:45:18.433403 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.433358    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d110d78c-27ee-4e76-9758-d92d9087d3a7-kube-api-access-5xg8v\" (UniqueName: \"kubernetes.io/projected/d110d78c-27ee-4e76-9758-d92d9087d3a7-kube-api-access-5xg8v\") pod \"d110d78c-27ee-4e76-9758-d92d9087d3a7\" (UID: \"d110d78c-27ee-4e76-9758-d92d9087d3a7\") "
Nov 03 21:45:18.433648 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.433628    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d110d78c-27ee-4e76-9758-d92d9087d3a7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d110d78c-27ee-4e76-9758-d92d9087d3a7-maas-api-service-ca\") pod \"d110d78c-27ee-4e76-9758-d92d9087d3a7\" (UID: \"d110d78c-27ee-4e76-9758-d92d9087d3a7\") "
Nov 03 21:45:18.433959 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.433935    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d110d78c-27ee-4e76-9758-d92d9087d3a7-maas-api-service-ca" pod "d110d78c-27ee-4e76-9758-d92d9087d3a7" (UID: "d110d78c-27ee-4e76-9758-d92d9087d3a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:18.434211 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.434089    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d110d78c-27ee-4e76-9758-d92d9087d3a7-maas-api-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:18.434211 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.434102    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsm2m\" (UniqueName: \"kubernetes.io/projected/b0cecdca-7f3c-4661-baac-6e3aefda5ce0-kube-api-access-dsm2m\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:18.435642 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.435621    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d110d78c-27ee-4e76-9758-d92d9087d3a7-kube-api-access-5xg8v" pod "d110d78c-27ee-4e76-9758-d92d9087d3a7" (UID: "d110d78c-27ee-4e76-9758-d92d9087d3a7"). InnerVolumeSpecName "kube-api-access-5xg8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:18.535335 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.535223    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xg8v\" (UniqueName: \"kubernetes.io/projected/d110d78c-27ee-4e76-9758-d92d9087d3a7-kube-api-access-5xg8v\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:18.992575 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.992539    5053 generic.go:356] "Generic (PLEG): container finished" podID="65569e2b-3978-464d-bec3-ecd8ad99836d" containerID="f56e4304ffb07f85c8e1915823d0478ffcb9189a1b3a74a5d34f2bc911d9831a" exitCode=60
Nov 03 21:45:18.993057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.992606    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm" event={"ID":"65569e2b-3978-464d-bec3-ecd8ad99836d","Type":"ContainerDied","Data":"f56e4304ffb07f85c8e1915823d0478ffcb9189a1b3a74a5d34f2bc911d9831a"}
Nov 03 21:45:18.993057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.992677    5053 scope.go:122] "RemoveContainer" containerID="ab4cdf84ab4dbd30fc1287536140775c9c2d8092121aebe673cecc02a4b763f8"
Nov 03 21:45:18.993177 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.993145    5053 scope.go:122] "RemoveContainer" containerID="f56e4304ffb07f85c8e1915823d0478ffcb9189a1b3a74a5d34f2bc911d9831a"
Nov 03 21:45:18.993584 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:18.993556    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm_odh-ai-gateway-infra(65569e2b-3978-464d-bec3-ecd8ad99836d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm" podUID="65569e2b-3978-464d-bec3-ecd8ad99836d"
Nov 03 21:45:18.994721 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.994699    5053 generic.go:356] "Generic (PLEG): container finished" podID="932934f1-361d-4768-b972-0a648503c085" containerID="23c014d927494a02b76d8d8ce5506326d3e491ddffc45908a37432f9c6c12eed" exitCode=60
Nov 03 21:45:18.994799 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.994747    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6" event={"ID":"932934f1-361d-4768-b972-0a648503c085","Type":"ContainerDied","Data":"23c014d927494a02b76d8d8ce5506326d3e491ddffc45908a37432f9c6c12eed"}
Nov 03 21:45:18.995165 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.995140    5053 scope.go:122] "RemoveContainer" containerID="23c014d927494a02b76d8d8ce5506326d3e491ddffc45908a37432f9c6c12eed"
Nov 03 21:45:18.995519 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:18.995468    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895705-kn9w6_odh-ai-gateway-infra(932934f1-361d-4768-b972-0a648503c085)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6" podUID="932934f1-361d-4768-b972-0a648503c085"
Nov 03 21:45:18.996143 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.996125    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq"
Nov 03 21:45:18.996254 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.996158    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq" event={"ID":"b0cecdca-7f3c-4661-baac-6e3aefda5ce0","Type":"ContainerDied","Data":"75ebbe92be9a1044bf6e3c347377366a626c1874238a61f6c5fe7c018751f574"}
Nov 03 21:45:18.998729 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.998711    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd"
Nov 03 21:45:18.998821 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:18.998713    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd" event={"ID":"d110d78c-27ee-4e76-9758-d92d9087d3a7","Type":"ContainerDied","Data":"67de70e727a853592a60ef8d5fdc6be2a822511dce737fcae3537693cc81c461"}
Nov 03 21:45:19.007863 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.007839    5053 scope.go:122] "RemoveContainer" containerID="66d3109ef5e3280066ab931102036d57abfe6b72729d29c4c4d843bd74121f70"
Nov 03 21:45:19.021917 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.021896    5053 scope.go:122] "RemoveContainer" containerID="6db4e97a5d1690f85d7d252d2a51cf2ac04be47f040f0ffa63d3b79fab47538a"
Nov 03 21:45:19.031892 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.031867    5053 scope.go:122] "RemoveContainer" containerID="408ff86d48584d96de5a1effe721c5d4959d7cf6be246ccc389dc03f75af6715"
Nov 03 21:45:19.219601 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.219560    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd"]
Nov 03 21:45:19.225646 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.225606    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895705-4svdd"]
Nov 03 21:45:19.331271 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.331204    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq"]
Nov 03 21:45:19.337396 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.337367    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895705-vrhxq"]
Nov 03 21:45:19.599021 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.598925    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"]
Nov 03 21:45:19.599920 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.599894    5053 cpu_manager.go:414] "containerMap: removing container" podUID="d110d78c-27ee-4e76-9758-d92d9087d3a7" containerName="cleanup"
Nov 03 21:45:19.599920 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.599918    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d110d78c-27ee-4e76-9758-d92d9087d3a7" containerName="cleanup"
Nov 03 21:45:19.600191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.599936    5053 cpu_manager.go:414] "containerMap: removing container" podUID="287cc853-6b9b-42c7-800e-6c0d8eca5d9e" containerName="cleanup"
Nov 03 21:45:19.600191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.599942    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="287cc853-6b9b-42c7-800e-6c0d8eca5d9e" containerName="cleanup"
Nov 03 21:45:19.600191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.599954    5053 cpu_manager.go:414] "containerMap: removing container" podUID="6a7b45d4-526f-4568-8036-1a614c502fe5" containerName="payload-pre-processing"
Nov 03 21:45:19.600191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.599959    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a7b45d4-526f-4568-8036-1a614c502fe5" containerName="payload-pre-processing"
Nov 03 21:45:19.600191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.599971    5053 cpu_manager.go:414] "containerMap: removing container" podUID="c02684d1-aaec-4345-9ce7-4d910ed4fbe9" containerName="payload-processing"
Nov 03 21:45:19.600191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.599976    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c02684d1-aaec-4345-9ce7-4d910ed4fbe9" containerName="payload-processing"
Nov 03 21:45:19.600191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.599983    5053 cpu_manager.go:414] "containerMap: removing container" podUID="b0cecdca-7f3c-4661-baac-6e3aefda5ce0" containerName="cleanup"
Nov 03 21:45:19.600191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.599987    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0cecdca-7f3c-4661-baac-6e3aefda5ce0" containerName="cleanup"
Nov 03 21:45:19.600191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.600003    5053 cpu_manager.go:414] "containerMap: removing container" podUID="b0cecdca-7f3c-4661-baac-6e3aefda5ce0" containerName="cleanup"
Nov 03 21:45:19.600191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.600007    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0cecdca-7f3c-4661-baac-6e3aefda5ce0" containerName="cleanup"
Nov 03 21:45:19.600191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.600017    5053 cpu_manager.go:414] "containerMap: removing container" podUID="287cc853-6b9b-42c7-800e-6c0d8eca5d9e" containerName="cleanup"
Nov 03 21:45:19.600191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.600053    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="287cc853-6b9b-42c7-800e-6c0d8eca5d9e" containerName="cleanup"
Nov 03 21:45:19.600191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.600064    5053 cpu_manager.go:414] "containerMap: removing container" podUID="d110d78c-27ee-4e76-9758-d92d9087d3a7" containerName="cleanup"
Nov 03 21:45:19.600191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.600071    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d110d78c-27ee-4e76-9758-d92d9087d3a7" containerName="cleanup"
Nov 03 21:45:19.600191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.600084    5053 cpu_manager.go:414] "containerMap: removing container" podUID="287cc853-6b9b-42c7-800e-6c0d8eca5d9e" containerName="cleanup"
Nov 03 21:45:19.600191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.600120    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="287cc853-6b9b-42c7-800e-6c0d8eca5d9e" containerName="cleanup"
Nov 03 21:45:19.600785 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.600560    5053 cpu_manager.go:414] "containerMap: removing container" podUID="b0cecdca-7f3c-4661-baac-6e3aefda5ce0" containerName="cleanup"
Nov 03 21:45:19.600785 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.600590    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0cecdca-7f3c-4661-baac-6e3aefda5ce0" containerName="cleanup"
Nov 03 21:45:19.600785 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.600723    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a7b45d4-526f-4568-8036-1a614c502fe5" containerName="payload-pre-processing"
Nov 03 21:45:19.600785 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.600740    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="c02684d1-aaec-4345-9ce7-4d910ed4fbe9" containerName="payload-processing"
Nov 03 21:45:19.600785 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.600747    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0cecdca-7f3c-4661-baac-6e3aefda5ce0" containerName="cleanup"
Nov 03 21:45:19.600785 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.600755    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="d110d78c-27ee-4e76-9758-d92d9087d3a7" containerName="cleanup"
Nov 03 21:45:19.600785 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.600760    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0cecdca-7f3c-4661-baac-6e3aefda5ce0" containerName="cleanup"
Nov 03 21:45:19.600785 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.600766    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="287cc853-6b9b-42c7-800e-6c0d8eca5d9e" containerName="cleanup"
Nov 03 21:45:19.600785 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.600775    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="d110d78c-27ee-4e76-9758-d92d9087d3a7" containerName="cleanup"
Nov 03 21:45:19.600785 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.600781    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="287cc853-6b9b-42c7-800e-6c0d8eca5d9e" containerName="cleanup"
Nov 03 21:45:19.606315 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.606293    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.677261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.674957    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="287cc853-6b9b-42c7-800e-6c0d8eca5d9e" path="/var/lib/kubelet/pods/287cc853-6b9b-42c7-800e-6c0d8eca5d9e/volumes"
Nov 03 21:45:19.677261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.675429    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0cecdca-7f3c-4661-baac-6e3aefda5ce0" path="/var/lib/kubelet/pods/b0cecdca-7f3c-4661-baac-6e3aefda5ce0/volumes"
Nov 03 21:45:19.677261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.675881    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d110d78c-27ee-4e76-9758-d92d9087d3a7" path="/var/lib/kubelet/pods/d110d78c-27ee-4e76-9758-d92d9087d3a7/volumes"
Nov 03 21:45:19.677261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.676708    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"]
Nov 03 21:45:19.750290 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.750257    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcjbd\" (UniqueName: \"kubernetes.io/projected/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-kube-api-access-dcjbd\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.750508 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.750327    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.750508 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.750422    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.750607 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.750528    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.750607 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.750592    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.750726 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.750630    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.750814 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.750793    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.750850 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.750832    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.750885 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.750860    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.851756 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.851665    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.851756 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.851748    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.851944 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.851807    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.851944 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.851862    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.852031 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.851956    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.852031 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.851991    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.852133 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.852046    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcjbd\" (UniqueName: \"kubernetes.io/projected/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-kube-api-access-dcjbd\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.852189 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.852136    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.852189 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.852167    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.852310 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.852191    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.852310 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.852209    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.852407 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.852325    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.852520 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.852492    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.852965 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.852912    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.854782 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.854754    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.855088 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.855066    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.908685 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.908658    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcjbd\" (UniqueName: \"kubernetes.io/projected/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-kube-api-access-dcjbd\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.908819 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.908701    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4s8qg\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:19.940890 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:19.940808    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:20.116930 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:20.116854    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm"]
Nov 03 21:45:20.300886 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:20.300862    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm"
Nov 03 21:45:20.380355 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:20.380326    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"]
Nov 03 21:45:20.381443 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:45:20.381416    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7bc971_f569_4f7c_88bf_5e3168bd49bc.slice/crio-3d6f6062457a2a033fa8e73b602d822be2ce3f663b476f8564afb645ac2df8d1 WatchSource:0}: Error finding container 3d6f6062457a2a033fa8e73b602d822be2ce3f663b476f8564afb645ac2df8d1: Status 404 returned error can't find the container with id 3d6f6062457a2a033fa8e73b602d822be2ce3f663b476f8564afb645ac2df8d1
Nov 03 21:45:20.384045 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:20.384012    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:45:20.384111 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:20.384078    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:45:20.384111 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:20.384107    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:45:20.462843 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:20.462809    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6"]
Nov 03 21:45:20.491073 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:20.490010    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65569e2b-3978-464d-bec3-ecd8ad99836d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65569e2b-3978-464d-bec3-ecd8ad99836d-maas-api-service-ca\") pod \"65569e2b-3978-464d-bec3-ecd8ad99836d\" (UID: \"65569e2b-3978-464d-bec3-ecd8ad99836d\") "
Nov 03 21:45:20.491073 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:20.490104    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65569e2b-3978-464d-bec3-ecd8ad99836d-kube-api-access-cmbgl\" (UniqueName: \"kubernetes.io/projected/65569e2b-3978-464d-bec3-ecd8ad99836d-kube-api-access-cmbgl\") pod \"65569e2b-3978-464d-bec3-ecd8ad99836d\" (UID: \"65569e2b-3978-464d-bec3-ecd8ad99836d\") "
Nov 03 21:45:20.491386 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:20.491346    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65569e2b-3978-464d-bec3-ecd8ad99836d-maas-api-service-ca" pod "65569e2b-3978-464d-bec3-ecd8ad99836d" (UID: "65569e2b-3978-464d-bec3-ecd8ad99836d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:20.494127 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:20.494079    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65569e2b-3978-464d-bec3-ecd8ad99836d-kube-api-access-cmbgl" pod "65569e2b-3978-464d-bec3-ecd8ad99836d" (UID: "65569e2b-3978-464d-bec3-ecd8ad99836d"). InnerVolumeSpecName "kube-api-access-cmbgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:20.592600 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:20.592566    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65569e2b-3978-464d-bec3-ecd8ad99836d-maas-api-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:20.592600 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:20.592601    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmbgl\" (UniqueName: \"kubernetes.io/projected/65569e2b-3978-464d-bec3-ecd8ad99836d-kube-api-access-cmbgl\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:20.629877 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:20.629799    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6"
Nov 03 21:45:20.795167 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:20.795130    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/932934f1-361d-4768-b972-0a648503c085-kube-api-access-lpkl7\" (UniqueName: \"kubernetes.io/projected/932934f1-361d-4768-b972-0a648503c085-kube-api-access-lpkl7\") pod \"932934f1-361d-4768-b972-0a648503c085\" (UID: \"932934f1-361d-4768-b972-0a648503c085\") "
Nov 03 21:45:20.795391 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:20.795272    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/932934f1-361d-4768-b972-0a648503c085-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932934f1-361d-4768-b972-0a648503c085-maas-api-service-ca\") pod \"932934f1-361d-4768-b972-0a648503c085\" (UID: \"932934f1-361d-4768-b972-0a648503c085\") "
Nov 03 21:45:20.795953 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:20.795918    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932934f1-361d-4768-b972-0a648503c085-maas-api-service-ca" pod "932934f1-361d-4768-b972-0a648503c085" (UID: "932934f1-361d-4768-b972-0a648503c085"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:20.796817 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:20.796795    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932934f1-361d-4768-b972-0a648503c085-maas-api-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:20.798172 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:20.798150    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932934f1-361d-4768-b972-0a648503c085-kube-api-access-lpkl7" pod "932934f1-361d-4768-b972-0a648503c085" (UID: "932934f1-361d-4768-b972-0a648503c085"). InnerVolumeSpecName "kube-api-access-lpkl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:20.897975 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:20.897876    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpkl7\" (UniqueName: \"kubernetes.io/projected/932934f1-361d-4768-b972-0a648503c085-kube-api-access-lpkl7\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:21.038595 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:21.038561    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm"
Nov 03 21:45:21.038777 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:21.038560    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm" event={"ID":"65569e2b-3978-464d-bec3-ecd8ad99836d","Type":"ContainerDied","Data":"bf33607c1348a0808a404ed8766d568bfc7906ede1f49c435ae38480d8de5e01"}
Nov 03 21:45:21.038777 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:21.038702    5053 scope.go:122] "RemoveContainer" containerID="f56e4304ffb07f85c8e1915823d0478ffcb9189a1b3a74a5d34f2bc911d9831a"
Nov 03 21:45:21.040543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:21.040515    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6"
Nov 03 21:45:21.040657 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:21.040554    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6" event={"ID":"932934f1-361d-4768-b972-0a648503c085","Type":"ContainerDied","Data":"5e017b89f65f985d758553b24c5dad1a0da3a66cc43ffae4a969d230634c54a9"}
Nov 03 21:45:21.042586 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:21.042557    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg" event={"ID":"5d7bc971-f569-4f7c-88bf-5e3168bd49bc","Type":"ContainerStarted","Data":"a7f3e462d4e398b3d733d20503cd3b342c84a2df3801b5f2782abe1ffc4cfde3"}
Nov 03 21:45:21.042689 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:21.042608    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg" event={"ID":"5d7bc971-f569-4f7c-88bf-5e3168bd49bc","Type":"ContainerStarted","Data":"3d6f6062457a2a033fa8e73b602d822be2ce3f663b476f8564afb645ac2df8d1"}
Nov 03 21:45:21.056988 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:21.056963    5053 scope.go:122] "RemoveContainer" containerID="23c014d927494a02b76d8d8ce5506326d3e491ddffc45908a37432f9c6c12eed"
Nov 03 21:45:21.227490 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:21.227266    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg" podStartSLOduration=2.227224801 podStartE2EDuration="2.227224801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:21.211744896 +0000 UTC m=+1835.705666769" watchObservedRunningTime="2026-11-03 21:45:21.227224801 +0000 UTC m=+1835.721146678"
Nov 03 21:45:21.332185 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:21.331432    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm"]
Nov 03 21:45:21.334974 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:21.334947    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895705-6zspm"]
Nov 03 21:45:21.378130 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:21.378098    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8"]
Nov 03 21:45:21.387778 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:21.387744    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85d7ff-4nlz8"]
Nov 03 21:45:21.448197 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:21.448168    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6"]
Nov 03 21:45:21.451158 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:21.451132    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895705-kn9w6"]
Nov 03 21:45:21.667455 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:21.667377    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65569e2b-3978-464d-bec3-ecd8ad99836d" path="/var/lib/kubelet/pods/65569e2b-3978-464d-bec3-ecd8ad99836d/volumes"
Nov 03 21:45:21.667848 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:21.667835    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="812811a5-95b8-4618-b7f1-2b807477a790" path="/var/lib/kubelet/pods/812811a5-95b8-4618-b7f1-2b807477a790/volumes"
Nov 03 21:45:21.668173 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:21.668161    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="932934f1-361d-4768-b972-0a648503c085" path="/var/lib/kubelet/pods/932934f1-361d-4768-b972-0a648503c085/volumes"
Nov 03 21:45:21.941483 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:21.941398    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:21.942893 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:21.942866    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-4s8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 03 21:45:21.943038 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:21.942924    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg" podUID="5d7bc971-f569-4f7c-88bf-5e3168bd49bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 03 21:45:22.941950 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:22.941914    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-4s8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 03 21:45:22.942452 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:22.941974    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg" podUID="5d7bc971-f569-4f7c-88bf-5e3168bd49bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 03 21:45:23.942316 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:23.942276    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-4s8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 03 21:45:23.942796 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:23.942332    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg" podUID="5d7bc971-f569-4f7c-88bf-5e3168bd49bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 03 21:45:24.941377 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:24.941339    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-4s8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 03 21:45:24.941652 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:24.941392    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg" podUID="5d7bc971-f569-4f7c-88bf-5e3168bd49bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 03 21:45:25.942041 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:25.941942    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-4s8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 03 21:45:25.942041 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:25.941997    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg" podUID="5d7bc971-f569-4f7c-88bf-5e3168bd49bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 03 21:45:26.941147 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:26.941111    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-4s8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 03 21:45:26.941373 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:26.941174    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg" podUID="5d7bc971-f569-4f7c-88bf-5e3168bd49bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 03 21:45:28.284270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:28.284199    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"]
Nov 03 21:45:28.284811 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:28.284406    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg" podUID="5a7feda9-6f61-4bf7-8561-05fec79b8b17" containerName="istio-proxy" containerID="cri-o://f257c7973b51a056eaaf58dcea26fab1f4aeb08b7a85ef74f65e7b38147f7cde" gracePeriod=30
Nov 03 21:45:28.739746 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:28.739706    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:28.740429 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:28.740401    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:28.741268 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:28.741222    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:45:29.830074 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.830022    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"]
Nov 03 21:45:29.831658 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.831627    5053 cpu_manager.go:414] "containerMap: removing container" podUID="932934f1-361d-4768-b972-0a648503c085" containerName="cleanup"
Nov 03 21:45:29.831658 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.831659    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="932934f1-361d-4768-b972-0a648503c085" containerName="cleanup"
Nov 03 21:45:29.831881 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.831696    5053 cpu_manager.go:414] "containerMap: removing container" podUID="932934f1-361d-4768-b972-0a648503c085" containerName="cleanup"
Nov 03 21:45:29.831881 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.831705    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="932934f1-361d-4768-b972-0a648503c085" containerName="cleanup"
Nov 03 21:45:29.831881 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.831742    5053 cpu_manager.go:414] "containerMap: removing container" podUID="65569e2b-3978-464d-bec3-ecd8ad99836d" containerName="cleanup"
Nov 03 21:45:29.831881 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.831749    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65569e2b-3978-464d-bec3-ecd8ad99836d" containerName="cleanup"
Nov 03 21:45:29.831881 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.831774    5053 cpu_manager.go:414] "containerMap: removing container" podUID="932934f1-361d-4768-b972-0a648503c085" containerName="cleanup"
Nov 03 21:45:29.831881 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.831781    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="932934f1-361d-4768-b972-0a648503c085" containerName="cleanup"
Nov 03 21:45:29.831881 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.831796    5053 cpu_manager.go:414] "containerMap: removing container" podUID="65569e2b-3978-464d-bec3-ecd8ad99836d" containerName="cleanup"
Nov 03 21:45:29.831881 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.831802    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65569e2b-3978-464d-bec3-ecd8ad99836d" containerName="cleanup"
Nov 03 21:45:29.831881 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.831819    5053 cpu_manager.go:414] "containerMap: removing container" podUID="d110d78c-27ee-4e76-9758-d92d9087d3a7" containerName="cleanup"
Nov 03 21:45:29.831881 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.831825    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d110d78c-27ee-4e76-9758-d92d9087d3a7" containerName="cleanup"
Nov 03 21:45:29.831881 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.831841    5053 cpu_manager.go:414] "containerMap: removing container" podUID="65569e2b-3978-464d-bec3-ecd8ad99836d" containerName="cleanup"
Nov 03 21:45:29.831881 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.831847    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65569e2b-3978-464d-bec3-ecd8ad99836d" containerName="cleanup"
Nov 03 21:45:29.832538 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.831982    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="932934f1-361d-4768-b972-0a648503c085" containerName="cleanup"
Nov 03 21:45:29.832538 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.832002    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0cecdca-7f3c-4661-baac-6e3aefda5ce0" containerName="cleanup"
Nov 03 21:45:29.832538 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.832014    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="932934f1-361d-4768-b972-0a648503c085" containerName="cleanup"
Nov 03 21:45:29.832538 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.832027    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="287cc853-6b9b-42c7-800e-6c0d8eca5d9e" containerName="cleanup"
Nov 03 21:45:29.832538 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.832039    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="65569e2b-3978-464d-bec3-ecd8ad99836d" containerName="cleanup"
Nov 03 21:45:29.832538 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.832050    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="d110d78c-27ee-4e76-9758-d92d9087d3a7" containerName="cleanup"
Nov 03 21:45:29.832538 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.832059    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="65569e2b-3978-464d-bec3-ecd8ad99836d" containerName="cleanup"
Nov 03 21:45:29.880692 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.880649    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:29.890895 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.890866    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"]
Nov 03 21:45:29.961354 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.961323    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istiod-ca-cert\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:29.961525 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.961369    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-credential-socket\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:29.961525 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.961412    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-workload-certs\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:29.961525 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.961431    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-data\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:29.961525 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.961472    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-envoy\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:29.961525 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.961512    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbdcc\" (UniqueName: \"kubernetes.io/projected/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-kube-api-access-jbdcc\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:29.961694 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.961546    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-podinfo\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:29.961694 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.961567    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-workload-socket\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:29.961694 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.961604    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-token\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:29.997646 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:29.997614    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-1c3bed-openshift-default-dockercfg-ggsrz\""
Nov 03 21:45:30.062638 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:30.062590    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-token\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:30.062848 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:30.062680    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istiod-ca-cert\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:30.062848 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:30.062732    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-credential-socket\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:30.062848 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:30.062800    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-workload-certs\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:30.062848 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:30.062827    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-data\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:30.062999 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:30.062901    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-envoy\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:30.062999 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:30.062970    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbdcc\" (UniqueName: \"kubernetes.io/projected/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-kube-api-access-jbdcc\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:30.063065 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:30.063024    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-podinfo\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:30.063101 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:30.063062    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-workload-socket\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:30.063688 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:30.063654    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istiod-ca-cert\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:30.063857 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:30.063826    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-credential-socket\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:30.064985 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:30.064962    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-workload-certs\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:30.065437 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:30.065415    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-data\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:30.067568 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:30.066479    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-podinfo\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:30.067568 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:30.067145    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-workload-socket\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:30.073030 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:30.072659    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-envoy\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:30.119905 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:30.119824    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbdcc\" (UniqueName: \"kubernetes.io/projected/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-kube-api-access-jbdcc\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:30.122350 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:30.122324    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-token\") pod \"e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") " pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:30.218724 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:30.218689    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:30.667273 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:30.667218    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"]
Nov 03 21:45:30.697661 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:45:30.697626    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc293ac5_37d8_4093_b8f4_5bdc2ba570bc.slice/crio-bc1726b1e5b3d404e468e4157d07ed952692b61c6f621cb64c6e96835d5a487d WatchSource:0}: Error finding container bc1726b1e5b3d404e468e4157d07ed952692b61c6f621cb64c6e96835d5a487d: Status 404 returned error can't find the container with id bc1726b1e5b3d404e468e4157d07ed952692b61c6f621cb64c6e96835d5a487d
Nov 03 21:45:30.700137 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:30.700107    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:45:30.700209 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:30.700171    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:45:30.700209 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:30.700197    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:45:31.117019 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:31.116983    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb" event={"ID":"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc","Type":"ContainerStarted","Data":"009c614ee0742a684a3fa093be169b8d2fa2e56d364ae582be3c6742c4091432"}
Nov 03 21:45:31.117505 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:31.117032    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb" event={"ID":"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc","Type":"ContainerStarted","Data":"bc1726b1e5b3d404e468e4157d07ed952692b61c6f621cb64c6e96835d5a487d"}
Nov 03 21:45:31.219411 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:31.219375    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:31.225067 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:31.225037    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:31.232635 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:31.232576    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb" podStartSLOduration=2.23256012 podStartE2EDuration="2.23256012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:31.231331198 +0000 UTC m=+1845.725253073" watchObservedRunningTime="2026-11-03 21:45:31.23256012 +0000 UTC m=+1845.726481995"
Nov 03 21:45:32.125591 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:32.125557    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:32.126748 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:32.126727    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:45:33.557631 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.557568    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:45:33.602556 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.602524    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-data\") pod \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") "
Nov 03 21:45:33.602719 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.602574    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istiod-ca-cert\") pod \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") "
Nov 03 21:45:33.602719 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.602608    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a7feda9-6f61-4bf7-8561-05fec79b8b17-kube-api-access-hrzkf\" (UniqueName: \"kubernetes.io/projected/5a7feda9-6f61-4bf7-8561-05fec79b8b17-kube-api-access-hrzkf\") pod \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") "
Nov 03 21:45:33.602719 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.602656    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-podinfo\") pod \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") "
Nov 03 21:45:33.602719 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.602691    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-token\" (UniqueName: \"kubernetes.io/projected/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-token\") pod \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") "
Nov 03 21:45:33.602924 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.602742    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-envoy\") pod \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") "
Nov 03 21:45:33.602924 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.602777    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-data" pod "5a7feda9-6f61-4bf7-8561-05fec79b8b17" (UID: "5a7feda9-6f61-4bf7-8561-05fec79b8b17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:45:33.602924 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.602796    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-workload-socket\") pod \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") "
Nov 03 21:45:33.602924 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.602837    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-workload-certs\") pod \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") "
Nov 03 21:45:33.602924 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.602889    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-credential-socket\") pod \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\" (UID: \"5a7feda9-6f61-4bf7-8561-05fec79b8b17\") "
Nov 03 21:45:33.603272 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.603217    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-workload-socket" pod "5a7feda9-6f61-4bf7-8561-05fec79b8b17" (UID: "5a7feda9-6f61-4bf7-8561-05fec79b8b17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:45:33.603403 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.603370    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-workload-certs" pod "5a7feda9-6f61-4bf7-8561-05fec79b8b17" (UID: "5a7feda9-6f61-4bf7-8561-05fec79b8b17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:45:33.603486 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.603453    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:33.603486 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.603475    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:33.603640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.603491    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-credential-socket" pod "5a7feda9-6f61-4bf7-8561-05fec79b8b17" (UID: "5a7feda9-6f61-4bf7-8561-05fec79b8b17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:45:33.603640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.603535    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istiod-ca-cert" pod "5a7feda9-6f61-4bf7-8561-05fec79b8b17" (UID: "5a7feda9-6f61-4bf7-8561-05fec79b8b17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:33.605636 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.605610    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-envoy" pod "5a7feda9-6f61-4bf7-8561-05fec79b8b17" (UID: "5a7feda9-6f61-4bf7-8561-05fec79b8b17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:45:33.605926 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.605900    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a7feda9-6f61-4bf7-8561-05fec79b8b17-kube-api-access-hrzkf" pod "5a7feda9-6f61-4bf7-8561-05fec79b8b17" (UID: "5a7feda9-6f61-4bf7-8561-05fec79b8b17"). InnerVolumeSpecName "kube-api-access-hrzkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:33.606034 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.605953    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-token" pod "5a7feda9-6f61-4bf7-8561-05fec79b8b17" (UID: "5a7feda9-6f61-4bf7-8561-05fec79b8b17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:33.606100 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.606076    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-podinfo" pod "5a7feda9-6f61-4bf7-8561-05fec79b8b17" (UID: "5a7feda9-6f61-4bf7-8561-05fec79b8b17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:45:33.705079 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.704995    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:33.705079 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.705029    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:33.705079 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.705044    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrzkf\" (UniqueName: \"kubernetes.io/projected/5a7feda9-6f61-4bf7-8561-05fec79b8b17-kube-api-access-hrzkf\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:33.705380 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.705060    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:33.705380 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.705118    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:33.705380 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.705132    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:33.705380 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:33.705144    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a7feda9-6f61-4bf7-8561-05fec79b8b17-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:34.142636 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:34.142596    5053 generic.go:356] "Generic (PLEG): container finished" podID="5a7feda9-6f61-4bf7-8561-05fec79b8b17" containerID="f257c7973b51a056eaaf58dcea26fab1f4aeb08b7a85ef74f65e7b38147f7cde" exitCode=0
Nov 03 21:45:34.142868 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:34.142662    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"
Nov 03 21:45:34.142868 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:34.142700    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg" event={"ID":"5a7feda9-6f61-4bf7-8561-05fec79b8b17","Type":"ContainerDied","Data":"f257c7973b51a056eaaf58dcea26fab1f4aeb08b7a85ef74f65e7b38147f7cde"}
Nov 03 21:45:34.142868 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:34.142763    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg" event={"ID":"5a7feda9-6f61-4bf7-8561-05fec79b8b17","Type":"ContainerDied","Data":"309469604e27229d8735536b81b57d81faea8e43056b095f3256497fec8ad1c2"}
Nov 03 21:45:34.142868 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:34.142791    5053 scope.go:122] "RemoveContainer" containerID="f257c7973b51a056eaaf58dcea26fab1f4aeb08b7a85ef74f65e7b38147f7cde"
Nov 03 21:45:34.153535 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:34.153515    5053 scope.go:122] "RemoveContainer" containerID="d817e5c652e1a0db2a2ea8d374f2d359ffb45317185a4a80e10c3efe9044d196"
Nov 03 21:45:34.163553 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:34.163533    5053 scope.go:122] "RemoveContainer" containerID="f257c7973b51a056eaaf58dcea26fab1f4aeb08b7a85ef74f65e7b38147f7cde"
Nov 03 21:45:34.163924 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:34.163900    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f257c7973b51a056eaaf58dcea26fab1f4aeb08b7a85ef74f65e7b38147f7cde\": container with ID starting with f257c7973b51a056eaaf58dcea26fab1f4aeb08b7a85ef74f65e7b38147f7cde not found: ID does not exist" containerID="f257c7973b51a056eaaf58dcea26fab1f4aeb08b7a85ef74f65e7b38147f7cde"
Nov 03 21:45:34.164023 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:34.163933    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f257c7973b51a056eaaf58dcea26fab1f4aeb08b7a85ef74f65e7b38147f7cde"} err="failed to get container status \"f257c7973b51a056eaaf58dcea26fab1f4aeb08b7a85ef74f65e7b38147f7cde\": rpc error: code = NotFound desc = could not find container \"f257c7973b51a056eaaf58dcea26fab1f4aeb08b7a85ef74f65e7b38147f7cde\": container with ID starting with f257c7973b51a056eaaf58dcea26fab1f4aeb08b7a85ef74f65e7b38147f7cde not found: ID does not exist"
Nov 03 21:45:34.164023 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:34.163950    5053 scope.go:122] "RemoveContainer" containerID="d817e5c652e1a0db2a2ea8d374f2d359ffb45317185a4a80e10c3efe9044d196"
Nov 03 21:45:34.164221 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:34.164200    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d817e5c652e1a0db2a2ea8d374f2d359ffb45317185a4a80e10c3efe9044d196\": container with ID starting with d817e5c652e1a0db2a2ea8d374f2d359ffb45317185a4a80e10c3efe9044d196 not found: ID does not exist" containerID="d817e5c652e1a0db2a2ea8d374f2d359ffb45317185a4a80e10c3efe9044d196"
Nov 03 21:45:34.164288 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:34.164229    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d817e5c652e1a0db2a2ea8d374f2d359ffb45317185a4a80e10c3efe9044d196"} err="failed to get container status \"d817e5c652e1a0db2a2ea8d374f2d359ffb45317185a4a80e10c3efe9044d196\": rpc error: code = NotFound desc = could not find container \"d817e5c652e1a0db2a2ea8d374f2d359ffb45317185a4a80e10c3efe9044d196\": container with ID starting with d817e5c652e1a0db2a2ea8d374f2d359ffb45317185a4a80e10c3efe9044d196 not found: ID does not exist"
Nov 03 21:45:34.306693 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:34.306660    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"]
Nov 03 21:45:34.311871 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:34.311844    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-85d7ff-openshift-default-5bcd994fff-bt5fg"]
Nov 03 21:45:35.668369 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:35.668335    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a7feda9-6f61-4bf7-8561-05fec79b8b17" path="/var/lib/kubelet/pods/5a7feda9-6f61-4bf7-8561-05fec79b8b17/volumes"
Nov 03 21:45:36.147989 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:36.147952    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"]
Nov 03 21:45:36.148856 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:36.148834    5053 cpu_manager.go:414] "containerMap: removing container" podUID="5a7feda9-6f61-4bf7-8561-05fec79b8b17" containerName="istio-proxy"
Nov 03 21:45:36.148929 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:36.148860    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a7feda9-6f61-4bf7-8561-05fec79b8b17" containerName="istio-proxy"
Nov 03 21:45:36.149054 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:36.149042    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a7feda9-6f61-4bf7-8561-05fec79b8b17" containerName="istio-proxy"
Nov 03 21:45:36.149094 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:36.149065    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="932934f1-361d-4768-b972-0a648503c085" containerName="cleanup"
Nov 03 21:45:36.149094 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:36.149079    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="65569e2b-3978-464d-bec3-ecd8ad99836d" containerName="cleanup"
Nov 03 21:45:36.154714 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:36.154691    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"
Nov 03 21:45:36.210220 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:36.210189    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"]
Nov 03 21:45:36.285849 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:36.265744    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv\" (UID: \"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"
Nov 03 21:45:36.285849 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:36.265862    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv\" (UID: \"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"
Nov 03 21:45:36.285849 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:36.265995    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv\" (UID: \"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"
Nov 03 21:45:36.285849 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:36.266050    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5b2c\" (UniqueName: \"kubernetes.io/projected/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-kube-api-access-m5b2c\") pod \"maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv\" (UID: \"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"
Nov 03 21:45:36.300271 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:36.300212    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-87e1cf-serving-cert\""
Nov 03 21:45:36.373484 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:36.373447    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv\" (UID: \"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"
Nov 03 21:45:36.373637 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:36.373557    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv\" (UID: \"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"
Nov 03 21:45:36.373695 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:36.373676    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv\" (UID: \"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"
Nov 03 21:45:36.373742 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:36.373725    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5b2c\" (UniqueName: \"kubernetes.io/projected/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-kube-api-access-m5b2c\") pod \"maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv\" (UID: \"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"
Nov 03 21:45:36.374906 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:36.374881    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv\" (UID: \"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"
Nov 03 21:45:36.375010 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:36.374995    5053 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-serving-cert: secret "maas-api-e2e-shared-a-w2-87e1cf-serving-cert" not found
Nov 03 21:45:36.375086 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:36.375077    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-maas-api-tls podName:cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3 nodeName:}" failed. No retries permitted until 2026-11-03 21:45:36.875059419 +0000 UTC m=+1851.368981278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-maas-api-tls") pod "maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv" (UID: "cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3") : secret "maas-api-e2e-shared-a-w2-87e1cf-serving-cert" not found
Nov 03 21:45:36.378523 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:36.378496    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv\" (UID: \"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"
Nov 03 21:45:36.436309 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:36.434330    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5b2c\" (UniqueName: \"kubernetes.io/projected/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-kube-api-access-m5b2c\") pod \"maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv\" (UID: \"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"
Nov 03 21:45:36.880227 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:36.880194    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv\" (UID: \"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"
Nov 03 21:45:36.882996 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:36.882947    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv\" (UID: \"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"
Nov 03 21:45:37.068501 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:37.068473    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"
Nov 03 21:45:37.463611 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:37.463573    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"]
Nov 03 21:45:37.469192 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:45:37.469140    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc3a1ac9_f959_4e0c_af45_4055ff2ec0d3.slice/crio-bb26703227aa566f37255201b5498a1d0d77def3502526d8909c2ea72f0651b3 WatchSource:0}: Error finding container bb26703227aa566f37255201b5498a1d0d77def3502526d8909c2ea72f0651b3: Status 404 returned error can't find the container with id bb26703227aa566f37255201b5498a1d0d77def3502526d8909c2ea72f0651b3
Nov 03 21:45:38.177799 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:38.177765    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv" event={"ID":"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3","Type":"ContainerStarted","Data":"bb26703227aa566f37255201b5498a1d0d77def3502526d8909c2ea72f0651b3"}
Nov 03 21:45:39.187322 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:39.187277    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv" event={"ID":"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3","Type":"ContainerStarted","Data":"6e497ff07b83e80e25bc48d7b673bc4f7daefd95f6a713785d9950f6d573fb1e"}
Nov 03 21:45:39.187731 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:39.187341    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"
Nov 03 21:45:39.304975 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:39.304908    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv" podStartSLOduration=2.481466047 podStartE2EDuration="3.30489281s" totalImagesPullingTime="823.426763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:39.301475891 +0000 UTC m=+1853.795397766" watchObservedRunningTime="2026-11-03 21:45:39.30489281 +0000 UTC m=+1853.798814685"
Nov 03 21:45:40.929758 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:45:40.929718    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:45:45.260300 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:45.260267    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"
Nov 03 21:45:47.071970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:47.071937    5053 scope.go:122] "RemoveContainer" containerID="39e7503a67875ad2c1c7b846a02997c55e5554593d4741c2073a54aa50abecd7"
Nov 03 21:45:47.082005 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:47.081980    5053 scope.go:122] "RemoveContainer" containerID="75bef1aaf411c95c4a183a16e79c7ec58110c4f066a7214e292a136bb239bedb"
Nov 03 21:45:52.218408 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.218372    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"]
Nov 03 21:45:52.219328 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.219310    5053 cpu_manager.go:414] "containerMap: removing container" podUID="5a7feda9-6f61-4bf7-8561-05fec79b8b17" containerName="istio-proxy"
Nov 03 21:45:52.219390 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.219332    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a7feda9-6f61-4bf7-8561-05fec79b8b17" containerName="istio-proxy"
Nov 03 21:45:52.219520 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.219508    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a7feda9-6f61-4bf7-8561-05fec79b8b17" containerName="istio-proxy"
Nov 03 21:45:52.225853 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.225824    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.278815 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.278771    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-3c6597-openshift-default-dockercfg-qjrx2\""
Nov 03 21:45:52.315970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.315375    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-workload-socket\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.315970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.315446    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-workload-certs\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.315970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.315511    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-credential-socket\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.315970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.315559    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-podinfo\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.315970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.315588    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-data\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.315970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.315618    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-envoy\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.315970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.315672    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-token\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.315970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.315745    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scvkq\" (UniqueName: \"kubernetes.io/projected/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-kube-api-access-scvkq\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.315970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.315777    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istiod-ca-cert\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.361084 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.360995    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"]
Nov 03 21:45:52.416597 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.416564    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scvkq\" (UniqueName: \"kubernetes.io/projected/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-kube-api-access-scvkq\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.416784 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.416606    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istiod-ca-cert\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.416784 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.416656    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-workload-socket\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.416784 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.416698    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-workload-certs\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.416784 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.416752    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-credential-socket\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.416992 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.416801    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-podinfo\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.416992 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.416828    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-data\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.416992 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.416847    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-envoy\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.416992 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.416887    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-token\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.417191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.417165    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-workload-certs\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.417572 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.417520    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-workload-socket\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.417704 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.417621    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-credential-socket\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.417704 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.417641    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-data\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.418344 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.418326    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istiod-ca-cert\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.419930 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.419909    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-envoy\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.421464 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.421445    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-podinfo\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.475942 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.475862    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-token\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.475942 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.475931    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scvkq\" (UniqueName: \"kubernetes.io/projected/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-kube-api-access-scvkq\") pod \"e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") " pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.597526 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.597485    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"]
Nov 03 21:45:52.601858 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.601834    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:52.602931 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.602916    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.699003 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.698964    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3c0bdb-openshift-default-dockercfg-w8nsl\""
Nov 03 21:45:52.736285 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.735037    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64lcw\" (UniqueName: \"kubernetes.io/projected/1dc20893-5839-4b09-b930-9b9f150105fa-kube-api-access-64lcw\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.736285 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.735085    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dc20893-5839-4b09-b930-9b9f150105fa-istiod-ca-cert\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.736285 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.735121    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dc20893-5839-4b09-b930-9b9f150105fa-istio-podinfo\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.736285 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.735149    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-workload-socket\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.736285 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.735254    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-credential-socket\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.736285 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.735325    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dc20893-5839-4b09-b930-9b9f150105fa-istio-token\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.736285 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.735350    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-istio-envoy\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.736285 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.735415    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-istio-data\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.736285 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.735487    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-workload-certs\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.793280 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.790536    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"]
Nov 03 21:45:52.838178 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.838134    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-istio-data\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.838413 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.838227    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-workload-certs\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.838413 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.838351    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64lcw\" (UniqueName: \"kubernetes.io/projected/1dc20893-5839-4b09-b930-9b9f150105fa-kube-api-access-64lcw\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.838413 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.838379    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dc20893-5839-4b09-b930-9b9f150105fa-istiod-ca-cert\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.838613 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.838413    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dc20893-5839-4b09-b930-9b9f150105fa-istio-podinfo\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.838613 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.838444    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-workload-socket\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.838613 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.838535    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-credential-socket\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.838613 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.838605    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dc20893-5839-4b09-b930-9b9f150105fa-istio-token\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.838827 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.838639    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-istio-envoy\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.860260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.839914    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dc20893-5839-4b09-b930-9b9f150105fa-istiod-ca-cert\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.860260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.840180    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-istio-data\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.860260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.840425    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-workload-certs\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.860260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.841124    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-workload-socket\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.860260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.859845    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-credential-socket\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.865342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.863805    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-istio-envoy\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.865342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.864580    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dc20893-5839-4b09-b930-9b9f150105fa-istio-podinfo\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.905081 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.903207    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64lcw\" (UniqueName: \"kubernetes.io/projected/1dc20893-5839-4b09-b930-9b9f150105fa-kube-api-access-64lcw\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.919620 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.919584    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dc20893-5839-4b09-b930-9b9f150105fa-istio-token\") pod \"e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") " pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:52.998934 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:52.998852    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:53.231505 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:53.231472    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"]
Nov 03 21:45:53.231858 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:45:53.231673    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f60ff6a_7db5_43cb_ac4c_c9d287139cf3.slice/crio-8eecca20b40767a06e2a91fc70d49ad2d3b2ae838b229bb80a6a4f23af1e671e WatchSource:0}: Error finding container 8eecca20b40767a06e2a91fc70d49ad2d3b2ae838b229bb80a6a4f23af1e671e: Status 404 returned error can't find the container with id 8eecca20b40767a06e2a91fc70d49ad2d3b2ae838b229bb80a6a4f23af1e671e
Nov 03 21:45:53.234186 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:53.234150    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:45:53.234291 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:53.234257    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:45:53.234336 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:53.234305    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:45:53.380570 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:53.380522    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd" event={"ID":"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3","Type":"ContainerStarted","Data":"80445fe3e688fc4de4fb5df38d984dc8c251ee1407fedc5c9e9bf675362a5a34"}
Nov 03 21:45:53.380570 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:53.380576    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd" event={"ID":"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3","Type":"ContainerStarted","Data":"8eecca20b40767a06e2a91fc70d49ad2d3b2ae838b229bb80a6a4f23af1e671e"}
Nov 03 21:45:53.396407 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:53.396379    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"]
Nov 03 21:45:53.397271 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:45:53.397211    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc20893_5839_4b09_b930_9b9f150105fa.slice/crio-ed7f8fd35ea18b915c76840b40c246b9fc0b6bf8cb6ddd565debc786c09c878c WatchSource:0}: Error finding container ed7f8fd35ea18b915c76840b40c246b9fc0b6bf8cb6ddd565debc786c09c878c: Status 404 returned error can't find the container with id ed7f8fd35ea18b915c76840b40c246b9fc0b6bf8cb6ddd565debc786c09c878c
Nov 03 21:45:53.399842 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:53.399810    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:45:53.399918 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:53.399878    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:45:53.399918 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:53.399908    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:45:53.499097 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:53.499045    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd" podStartSLOduration=1.499029693 podStartE2EDuration="1.499029693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:53.49456715 +0000 UTC m=+1867.988489017" watchObservedRunningTime="2026-11-03 21:45:53.499029693 +0000 UTC m=+1867.992951559"
Nov 03 21:45:53.602389 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:53.602358    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:54.391128 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:54.391091    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2" event={"ID":"1dc20893-5839-4b09-b930-9b9f150105fa","Type":"ContainerStarted","Data":"35b65990e60f7449e382b9e2e3b20ffef53706f86b55874c33f94d6cd0dd9a51"}
Nov 03 21:45:54.391560 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:54.391197    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2" event={"ID":"1dc20893-5839-4b09-b930-9b9f150105fa","Type":"ContainerStarted","Data":"ed7f8fd35ea18b915c76840b40c246b9fc0b6bf8cb6ddd565debc786c09c878c"}
Nov 03 21:45:54.560846 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:54.560796    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2" podStartSLOduration=2.560782337 podStartE2EDuration="2.560782337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:54.557113496 +0000 UTC m=+1869.051035369" watchObservedRunningTime="2026-11-03 21:45:54.560782337 +0000 UTC m=+1869.054704211"
Nov 03 21:45:54.609523 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:54.609495    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:54.999773 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:54.999741    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:55.006354 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:55.006330    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:55.400554 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:55.400466    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:45:55.401075 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:55.401051    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:55.401304 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:55.401278    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:45:55.401516 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:45:55.401499    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:46:06.224459 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:06.224418    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"]
Nov 03 21:46:06.225020 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:06.224683    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9" podUID="9bf836a1-d2b2-4f9b-bca1-92a40c71811b" containerName="maas-api" containerID="cri-o://7bcdf3debbd77a98cfbde34c3c3d3dfa5e942b7d02128408729d3ed93e832a7d" gracePeriod=30
Nov 03 21:46:06.499321 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:06.499189    5053 generic.go:356] "Generic (PLEG): container finished" podID="9bf836a1-d2b2-4f9b-bca1-92a40c71811b" containerID="7bcdf3debbd77a98cfbde34c3c3d3dfa5e942b7d02128408729d3ed93e832a7d" exitCode=0
Nov 03 21:46:06.499321 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:06.499305    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9" event={"ID":"9bf836a1-d2b2-4f9b-bca1-92a40c71811b","Type":"ContainerDied","Data":"7bcdf3debbd77a98cfbde34c3c3d3dfa5e942b7d02128408729d3ed93e832a7d"}
Nov 03 21:46:06.545359 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:06.545331    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"
Nov 03 21:46:06.599606 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:06.599561    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-maas-api-tls\") pod \"9bf836a1-d2b2-4f9b-bca1-92a40c71811b\" (UID: \"9bf836a1-d2b2-4f9b-bca1-92a40c71811b\") "
Nov 03 21:46:06.599812 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:06.599664    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-openshift-service-ca\") pod \"9bf836a1-d2b2-4f9b-bca1-92a40c71811b\" (UID: \"9bf836a1-d2b2-4f9b-bca1-92a40c71811b\") "
Nov 03 21:46:06.599812 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:06.599698    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-kube-api-access-2zcbh\" (UniqueName: \"kubernetes.io/projected/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-kube-api-access-2zcbh\") pod \"9bf836a1-d2b2-4f9b-bca1-92a40c71811b\" (UID: \"9bf836a1-d2b2-4f9b-bca1-92a40c71811b\") "
Nov 03 21:46:06.599943 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:06.599876    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-metrics-certs\") pod \"9bf836a1-d2b2-4f9b-bca1-92a40c71811b\" (UID: \"9bf836a1-d2b2-4f9b-bca1-92a40c71811b\") "
Nov 03 21:46:06.600114 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:06.600087    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-openshift-service-ca" pod "9bf836a1-d2b2-4f9b-bca1-92a40c71811b" (UID: "9bf836a1-d2b2-4f9b-bca1-92a40c71811b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:46:06.600520 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:06.600446    5053 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-openshift-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:06.604951 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:06.604921    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-metrics-certs" pod "9bf836a1-d2b2-4f9b-bca1-92a40c71811b" (UID: "9bf836a1-d2b2-4f9b-bca1-92a40c71811b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:46:06.620805 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:06.612272    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-kube-api-access-2zcbh" pod "9bf836a1-d2b2-4f9b-bca1-92a40c71811b" (UID: "9bf836a1-d2b2-4f9b-bca1-92a40c71811b"). InnerVolumeSpecName "kube-api-access-2zcbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:46:06.620805 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:06.617929    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-maas-api-tls" pod "9bf836a1-d2b2-4f9b-bca1-92a40c71811b" (UID: "9bf836a1-d2b2-4f9b-bca1-92a40c71811b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:46:06.702121 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:06.702083    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zcbh\" (UniqueName: \"kubernetes.io/projected/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-kube-api-access-2zcbh\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:06.702354 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:06.702173    5053 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-metrics-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:06.702354 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:06.702193    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf836a1-d2b2-4f9b-bca1-92a40c71811b-maas-api-tls\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:07.509951 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:07.509917    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"
Nov 03 21:46:07.509951 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:07.509936    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9" event={"ID":"9bf836a1-d2b2-4f9b-bca1-92a40c71811b","Type":"ContainerDied","Data":"0e41bd9a98200b96723d03e4cf342af0bed1c693b231f300d3d28ef9943007a6"}
Nov 03 21:46:07.511012 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:07.510016    5053 scope.go:122] "RemoveContainer" containerID="7bcdf3debbd77a98cfbde34c3c3d3dfa5e942b7d02128408729d3ed93e832a7d"
Nov 03 21:46:07.673719 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:07.673685    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"]
Nov 03 21:46:07.723963 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:07.723930    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-89ddad-6f6d64ffdf-dcfg9"]
Nov 03 21:46:08.330942 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:08.330905    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc"]
Nov 03 21:46:08.378668 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:08.378638    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-89ddad-xvxhc"]
Nov 03 21:46:09.667389 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:09.667357    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bf836a1-d2b2-4f9b-bca1-92a40c71811b" path="/var/lib/kubelet/pods/9bf836a1-d2b2-4f9b-bca1-92a40c71811b/volumes"
Nov 03 21:46:09.667820 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:09.667806    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d64bdf87-f4f6-4015-9d08-f7d4feeb18fc" path="/var/lib/kubelet/pods/d64bdf87-f4f6-4015-9d08-f7d4feeb18fc/volumes"
Nov 03 21:46:10.929764 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:46:10.929728    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:46:11.603321 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.603291    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"]
Nov 03 21:46:11.603898 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.603883    5053 cpu_manager.go:414] "containerMap: removing container" podUID="9bf836a1-d2b2-4f9b-bca1-92a40c71811b" containerName="maas-api"
Nov 03 21:46:11.603945 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.603899    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bf836a1-d2b2-4f9b-bca1-92a40c71811b" containerName="maas-api"
Nov 03 21:46:11.604043 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.604034    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bf836a1-d2b2-4f9b-bca1-92a40c71811b" containerName="maas-api"
Nov 03 21:46:11.609803 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.609765    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.653419 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.653389    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"]
Nov 03 21:46:11.657596 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.657576    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.675887 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.675859    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-token\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.676024 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.675922    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-data\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.676074 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.676039    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-podinfo\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.676074 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.676069    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-workload-socket\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.676143 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.676090    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-envoy\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.676193 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.676176    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istiod-ca-cert\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.676231 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.676203    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-credential-socket\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.676306 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.676271    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-workload-certs\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.676340 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.676313    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6p7w\" (UniqueName: \"kubernetes.io/projected/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-kube-api-access-m6p7w\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.677336 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.677315    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-87e1cf-openshift-default-dockercfg-28lnl\""
Nov 03 21:46:11.708143 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.708113    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"]
Nov 03 21:46:11.762730 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.762646    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"]
Nov 03 21:46:11.777725 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.777685    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-envoy\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.777931 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.777763    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istiod-ca-cert\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.777931 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.777798    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-credential-socket\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.777931 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.777839    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-data\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.777931 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.777904    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-workload-certs\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.777931 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.777931    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6p7w\" (UniqueName: \"kubernetes.io/projected/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-kube-api-access-m6p7w\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.778208 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.777967    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-credential-socket\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.778208 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.778008    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-workload-socket\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.778208 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.778032    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-workload-certs\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.778208 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.778079    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-token\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.778530 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.778501    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-workload-certs\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.778807 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.778759    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istiod-ca-cert\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.778967 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.778865    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-envoy\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.778967 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.778904    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-podinfo\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.779100 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.778965    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-token\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.779100 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.778988    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-credential-socket\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.779100 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.779071    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-data\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.779281 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.779127    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhdvg\" (UniqueName: \"kubernetes.io/projected/ccf8ba47-571b-4ad8-ae83-f035c92f485a-kube-api-access-hhdvg\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.779281 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.779220    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-podinfo\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.779281 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.779265    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-workload-socket\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.779659 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.779571    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-data\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.779659 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.779614    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istiod-ca-cert\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.779799 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.779651    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-workload-socket\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.780852 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.780830    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-envoy\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.782346 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.782319    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-podinfo\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.836176 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.836138    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6p7w\" (UniqueName: \"kubernetes.io/projected/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-kube-api-access-m6p7w\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.836496 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.836471    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-token\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.880985 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.880893    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-podinfo\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.880985 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.880983    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhdvg\" (UniqueName: \"kubernetes.io/projected/ccf8ba47-571b-4ad8-ae83-f035c92f485a-kube-api-access-hhdvg\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.881194 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.881058    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-data\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.881194 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.881119    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-credential-socket\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.881194 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.881153    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-workload-socket\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.881194 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.881174    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-workload-certs\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.881424 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.881213    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-token\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.881424 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.881322    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istiod-ca-cert\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.881424 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.881378    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-envoy\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.881922 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.881817    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-data\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.881922 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.881904    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-credential-socket\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.882091 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.882031    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-workload-certs\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.882153 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.882126    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-workload-socket\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.882459 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.882434    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istiod-ca-cert\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.883825 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.883801    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-envoy\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.883930 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.883848    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-podinfo\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.928123 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.928087    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:11.939125 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.939097    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-token\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.939565 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.939429    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhdvg\" (UniqueName: \"kubernetes.io/projected/ccf8ba47-571b-4ad8-ae83-f035c92f485a-kube-api-access-hhdvg\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:11.971781 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:11.971742    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:12.363210 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:12.363175    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"]
Nov 03 21:46:12.399487 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:46:12.399456    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda570d7ee_ec67_4db1_a2d2_0ffc18b0d439.slice/crio-b1c7624a79882f78e057ec9ce5ffe66fef7f60103e8c41c5c3c0bf95c59b060a WatchSource:0}: Error finding container b1c7624a79882f78e057ec9ce5ffe66fef7f60103e8c41c5c3c0bf95c59b060a: Status 404 returned error can't find the container with id b1c7624a79882f78e057ec9ce5ffe66fef7f60103e8c41c5c3c0bf95c59b060a
Nov 03 21:46:12.402619 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:12.402584    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:46:12.402708 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:12.402658    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:46:12.402708 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:12.402686    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:46:12.412449 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:12.412423    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"]
Nov 03 21:46:12.413696 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:46:12.413674    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf8ba47_571b_4ad8_ae83_f035c92f485a.slice/crio-142759b9bc924a15e9b9e8f26aba1878b4119730700b921ebed01f07c2a0f4f9 WatchSource:0}: Error finding container 142759b9bc924a15e9b9e8f26aba1878b4119730700b921ebed01f07c2a0f4f9: Status 404 returned error can't find the container with id 142759b9bc924a15e9b9e8f26aba1878b4119730700b921ebed01f07c2a0f4f9
Nov 03 21:46:12.416417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:12.416384    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:46:12.416517 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:12.416452    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:46:12.416517 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:12.416481    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:46:12.558258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:12.557935    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96" event={"ID":"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439","Type":"ContainerStarted","Data":"b1789c94394ea93bdcee8f0efdee483e716e93e3a82edc31ab4d72826b329685"}
Nov 03 21:46:12.558258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:12.558002    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96" event={"ID":"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439","Type":"ContainerStarted","Data":"b1c7624a79882f78e057ec9ce5ffe66fef7f60103e8c41c5c3c0bf95c59b060a"}
Nov 03 21:46:12.564267 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:12.563730    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89" event={"ID":"ccf8ba47-571b-4ad8-ae83-f035c92f485a","Type":"ContainerStarted","Data":"4da43c0f2ad530ed886d5993a626837cc1c280c140e2546bc159caa9232ad905"}
Nov 03 21:46:12.564267 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:12.563790    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89" event={"ID":"ccf8ba47-571b-4ad8-ae83-f035c92f485a","Type":"ContainerStarted","Data":"142759b9bc924a15e9b9e8f26aba1878b4119730700b921ebed01f07c2a0f4f9"}
Nov 03 21:46:12.690223 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:12.690111    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96" podStartSLOduration=1.690093272 podStartE2EDuration="1.690093272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:12.689573976 +0000 UTC m=+1887.183495854" watchObservedRunningTime="2026-11-03 21:46:12.690093272 +0000 UTC m=+1887.184015147"
Nov 03 21:46:12.814259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:12.806203    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89" podStartSLOduration=1.8061818120000002 podStartE2EDuration="1.806181812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:12.80573287 +0000 UTC m=+1887.299654745" watchObservedRunningTime="2026-11-03 21:46:12.806181812 +0000 UTC m=+1887.300103682"
Nov 03 21:46:12.929147 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:12.929107    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:12.973719 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:12.972331    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:13.950430 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:13.950381    5053 patch_prober.go:29] interesting pod/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 21:46:13.950738 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:13.950706    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96" podUID="a570d7ee-ec67-4db1-a2d2-0ffc18b0d439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 21:46:13.973872 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:13.973833    5053 patch_prober.go:29] interesting pod/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 03 21:46:13.974367 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:13.974327    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89" podUID="ccf8ba47-571b-4ad8-ae83-f035c92f485a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 03 21:46:14.940260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:14.929725    5053 patch_prober.go:29] interesting pod/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 21:46:14.940260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:14.929791    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96" podUID="a570d7ee-ec67-4db1-a2d2-0ffc18b0d439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 21:46:14.972555 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:14.972519    5053 patch_prober.go:29] interesting pod/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 03 21:46:14.972812 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:14.972785    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89" podUID="ccf8ba47-571b-4ad8-ae83-f035c92f485a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 03 21:46:15.596066 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:15.596032    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"]
Nov 03 21:46:15.596573 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:15.596283    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt" podUID="46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" containerName="istio-proxy" containerID="cri-o://8120e55e4dbee02ec044a208e8739b9815107a0696f315f354ca5f5cf073cad3" gracePeriod=30
Nov 03 21:46:15.932321 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:15.931475    5053 patch_prober.go:29] interesting pod/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 21:46:15.932321 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:15.931538    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96" podUID="a570d7ee-ec67-4db1-a2d2-0ffc18b0d439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 21:46:15.972870 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:15.972833    5053 patch_prober.go:29] interesting pod/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 03 21:46:15.973067 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:15.972925    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89" podUID="ccf8ba47-571b-4ad8-ae83-f035c92f485a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 03 21:46:16.934040 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:16.934006    5053 patch_prober.go:29] interesting pod/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 21:46:16.934681 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:16.934652    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96" podUID="a570d7ee-ec67-4db1-a2d2-0ffc18b0d439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 21:46:16.972955 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:16.972926    5053 patch_prober.go:29] interesting pod/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 03 21:46:16.973182 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:16.973162    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89" podUID="ccf8ba47-571b-4ad8-ae83-f035c92f485a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 03 21:46:17.932683 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:17.932646    5053 patch_prober.go:29] interesting pod/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 21:46:17.958389 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:17.958338    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96" podUID="a570d7ee-ec67-4db1-a2d2-0ffc18b0d439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 21:46:17.974658 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:17.974628    5053 patch_prober.go:29] interesting pod/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 03 21:46:17.974887 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:17.974866    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89" podUID="ccf8ba47-571b-4ad8-ae83-f035c92f485a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 03 21:46:18.933428 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:18.933396    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:18.933905 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:18.933878    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:18.934713 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:18.934693    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:46:18.977271 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:18.977209    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:18.977662 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:18.977534    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:18.978231 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:18.978204    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 21:46:20.715638 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:20.715616    5053 generic.go:356] "Generic (PLEG): container finished" podID="46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" containerID="8120e55e4dbee02ec044a208e8739b9815107a0696f315f354ca5f5cf073cad3" exitCode=0
Nov 03 21:46:20.715996 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:20.715692    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt" event={"ID":"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c","Type":"ContainerDied","Data":"8120e55e4dbee02ec044a208e8739b9815107a0696f315f354ca5f5cf073cad3"}
Nov 03 21:46:20.715996 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:20.715776    5053 scope.go:122] "RemoveContainer" containerID="ea2b8ae0e04cdd2189d3750064ed76bebcdc91dac800f1fa3721b6d91000fe17"
Nov 03 21:46:20.933408 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:20.933379    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:46:21.070431 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.070395    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-podinfo\") pod \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") "
Nov 03 21:46:21.070640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.070592    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-workload-socket\") pod \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") "
Nov 03 21:46:21.070640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.070625    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-envoy\") pod \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") "
Nov 03 21:46:21.070766 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.070646    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-token\" (UniqueName: \"kubernetes.io/projected/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-token\") pod \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") "
Nov 03 21:46:21.070766 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.070747    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-kube-api-access-5kmxh\" (UniqueName: \"kubernetes.io/projected/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-kube-api-access-5kmxh\") pod \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") "
Nov 03 21:46:21.070864 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.070800    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istiod-ca-cert\") pod \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") "
Nov 03 21:46:21.070864 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.070834    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-workload-certs\") pod \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") "
Nov 03 21:46:21.070971 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.070870    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-data\") pod \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") "
Nov 03 21:46:21.070971 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.070926    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-workload-socket" pod "46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" (UID: "46d4ed21-ed2d-40ea-ad11-24a268d5ef0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:46:21.070971 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.070935    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-credential-socket\") pod \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\" (UID: \"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c\") "
Nov 03 21:46:21.071221 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.071115    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-workload-certs" pod "46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" (UID: "46d4ed21-ed2d-40ea-ad11-24a268d5ef0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:46:21.071221 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.071185    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istiod-ca-cert" pod "46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" (UID: "46d4ed21-ed2d-40ea-ad11-24a268d5ef0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:46:21.071457 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.071345    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-credential-socket" pod "46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" (UID: "46d4ed21-ed2d-40ea-ad11-24a268d5ef0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:46:21.071457 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.071379    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-data" pod "46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" (UID: "46d4ed21-ed2d-40ea-ad11-24a268d5ef0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:46:21.071633 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.071612    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:21.071725 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.071635    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:21.071725 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.071649    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:21.071725 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.071661    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:21.071725 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.071673    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:21.073579 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.073552    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-envoy" pod "46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" (UID: "46d4ed21-ed2d-40ea-ad11-24a268d5ef0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:46:21.073654 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.073578    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-podinfo" pod "46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" (UID: "46d4ed21-ed2d-40ea-ad11-24a268d5ef0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:46:21.073654 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.073627    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-kube-api-access-5kmxh" pod "46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" (UID: "46d4ed21-ed2d-40ea-ad11-24a268d5ef0c"). InnerVolumeSpecName "kube-api-access-5kmxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:46:21.073724 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.073657    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-token" pod "46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" (UID: "46d4ed21-ed2d-40ea-ad11-24a268d5ef0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:46:21.172883 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.172788    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:21.172883 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.172828    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:21.172883 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.172845    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:21.172883 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.172854    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kmxh\" (UniqueName: \"kubernetes.io/projected/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c-kube-api-access-5kmxh\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:21.724956 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.724916    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt" event={"ID":"46d4ed21-ed2d-40ea-ad11-24a268d5ef0c","Type":"ContainerDied","Data":"ef3ca9fd664cb2db27ea3fac16aeda0be223e84e5413cf854b15891f288d7bb8"}
Nov 03 21:46:21.724956 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.724942    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"
Nov 03 21:46:21.725563 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.724985    5053 scope.go:122] "RemoveContainer" containerID="8120e55e4dbee02ec044a208e8739b9815107a0696f315f354ca5f5cf073cad3"
Nov 03 21:46:21.891086 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.891050    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"]
Nov 03 21:46:21.940513 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:21.940422    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-89ddad-openshift-default-5bbb8b5c99-vlmmt"]
Nov 03 21:46:23.668689 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:23.668650    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" path="/var/lib/kubelet/pods/46d4ed21-ed2d-40ea-ad11-24a268d5ef0c/volumes"
Nov 03 21:46:35.068108 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.068068    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"]
Nov 03 21:46:35.068860 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.068840    5053 cpu_manager.go:414] "containerMap: removing container" podUID="46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" containerName="istio-proxy"
Nov 03 21:46:35.068925 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.068862    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" containerName="istio-proxy"
Nov 03 21:46:35.068925 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.068892    5053 cpu_manager.go:414] "containerMap: removing container" podUID="46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" containerName="istio-proxy"
Nov 03 21:46:35.068925 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.068897    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" containerName="istio-proxy"
Nov 03 21:46:35.069017 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.068996    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" containerName="istio-proxy"
Nov 03 21:46:35.076336 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.075138    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.106184 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.106127    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-workload-certs\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.106412 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.106193    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-podinfo\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.106412 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.106223    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-envoy\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.106412 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.106296    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istiod-ca-cert\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.106412 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.106374    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jbpl\" (UniqueName: \"kubernetes.io/projected/d17d1dfe-8556-43d4-b67f-d0e628552f0b-kube-api-access-5jbpl\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.106652 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.106415    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-token\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.106652 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.106503    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-workload-socket\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.106652 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.106545    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-credential-socket\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.106652 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.106571    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-data\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.135286 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.135251    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-6ddbba-openshift-default-dockercfg-4vrj6\""
Nov 03 21:46:35.238859 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.235656    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-workload-certs\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.240907 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.240876    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-podinfo\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.241117 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.241101    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-envoy\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.241322 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.241303    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istiod-ca-cert\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.241533 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.241517    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jbpl\" (UniqueName: \"kubernetes.io/projected/d17d1dfe-8556-43d4-b67f-d0e628552f0b-kube-api-access-5jbpl\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.241663 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.241650    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-token\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.241858 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.241836    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-workload-socket\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.241987 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.241973    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-credential-socket\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.242094 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.242080    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-data\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.242654 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.242619    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-data\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.250831 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.250101    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istiod-ca-cert\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.250831 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.250415    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-workload-certs\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.252668 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.252103    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-workload-socket\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.252668 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.252374    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-credential-socket\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.260424 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.260302    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-envoy\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.276646 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.276203    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"]
Nov 03 21:46:35.278079 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.278020    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-podinfo\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.319653 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.312617    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-token\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.344391 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.344324    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jbpl\" (UniqueName: \"kubernetes.io/projected/d17d1dfe-8556-43d4-b67f-d0e628552f0b-kube-api-access-5jbpl\") pod \"e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") " pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.393712 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.393685    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:35.829638 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:46:35.829591    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17d1dfe_8556_43d4_b67f_d0e628552f0b.slice/crio-7bed2da7de8ab50f1aed39bf03c0ff3f46c73af91fdb6e09246c297a8bbf1e3e WatchSource:0}: Error finding container 7bed2da7de8ab50f1aed39bf03c0ff3f46c73af91fdb6e09246c297a8bbf1e3e: Status 404 returned error can't find the container with id 7bed2da7de8ab50f1aed39bf03c0ff3f46c73af91fdb6e09246c297a8bbf1e3e
Nov 03 21:46:35.829822 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.829795    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"]
Nov 03 21:46:35.832788 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.832749    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:46:35.832882 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.832833    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:46:35.832923 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.832908    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:46:35.867025 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:35.866988    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs" event={"ID":"d17d1dfe-8556-43d4-b67f-d0e628552f0b","Type":"ContainerStarted","Data":"7bed2da7de8ab50f1aed39bf03c0ff3f46c73af91fdb6e09246c297a8bbf1e3e"}
Nov 03 21:46:36.875180 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:36.875143    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs" event={"ID":"d17d1dfe-8556-43d4-b67f-d0e628552f0b","Type":"ContainerStarted","Data":"adc7e544a6d458cdd2ef0b533d07f79cdca60a608588c54144966b1b578bed41"}
Nov 03 21:46:37.044173 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:37.044110    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs" podStartSLOduration=2.044090634 podStartE2EDuration="2.044090634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:37.041133773 +0000 UTC m=+1911.535055659" watchObservedRunningTime="2026-11-03 21:46:37.044090634 +0000 UTC m=+1911.538012511"
Nov 03 21:46:37.393982 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:37.393950    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:37.399705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:37.399683    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:37.883227 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:37.883197    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:37.884305 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:37.884283    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:46:40.928873 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:46:40.928840    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:46:47.156714 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:47.156674    5053 scope.go:122] "RemoveContainer" containerID="0d1bf81374affda12d88632382413e57c6cce38eb93e88ba1c6f562867acdeff"
Nov 03 21:46:47.525045 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:47.524998    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"]
Nov 03 21:46:47.525435 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:47.525400    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs" podUID="1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344" containerName="istio-proxy" containerID="cri-o://55cfe9759b5da932271ca0a43b7092fc4b245a3a3f5f68f60ac730d93a6c7985" gracePeriod=30
Nov 03 21:46:49.578215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:49.578172    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"]
Nov 03 21:46:49.578980 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:49.578962    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d4ed21-ed2d-40ea-ad11-24a268d5ef0c" containerName="istio-proxy"
Nov 03 21:46:49.584197 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:49.584178    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"
Nov 03 21:46:49.636500 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:49.636471    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3c0bdb-serving-cert\""
Nov 03 21:46:49.640260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:49.640215    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"]
Nov 03 21:46:49.745774 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:49.745734    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/986245b5-a0a6-4997-92aa-3b1da76ac041-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7\" (UID: \"986245b5-a0a6-4997-92aa-3b1da76ac041\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"
Nov 03 21:46:49.746194 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:49.746173    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/986245b5-a0a6-4997-92aa-3b1da76ac041-metrics-certs\") pod \"maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7\" (UID: \"986245b5-a0a6-4997-92aa-3b1da76ac041\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"
Nov 03 21:46:49.746323 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:49.746271    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g2nx\" (UniqueName: \"kubernetes.io/projected/986245b5-a0a6-4997-92aa-3b1da76ac041-kube-api-access-8g2nx\") pod \"maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7\" (UID: \"986245b5-a0a6-4997-92aa-3b1da76ac041\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"
Nov 03 21:46:49.746391 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:49.746331    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/986245b5-a0a6-4997-92aa-3b1da76ac041-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7\" (UID: \"986245b5-a0a6-4997-92aa-3b1da76ac041\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"
Nov 03 21:46:49.872854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:49.869592    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/986245b5-a0a6-4997-92aa-3b1da76ac041-metrics-certs\") pod \"maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7\" (UID: \"986245b5-a0a6-4997-92aa-3b1da76ac041\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"
Nov 03 21:46:49.872854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:49.869670    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g2nx\" (UniqueName: \"kubernetes.io/projected/986245b5-a0a6-4997-92aa-3b1da76ac041-kube-api-access-8g2nx\") pod \"maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7\" (UID: \"986245b5-a0a6-4997-92aa-3b1da76ac041\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"
Nov 03 21:46:49.872854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:49.869714    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/986245b5-a0a6-4997-92aa-3b1da76ac041-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7\" (UID: \"986245b5-a0a6-4997-92aa-3b1da76ac041\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"
Nov 03 21:46:49.872854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:49.869798    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/986245b5-a0a6-4997-92aa-3b1da76ac041-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7\" (UID: \"986245b5-a0a6-4997-92aa-3b1da76ac041\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"
Nov 03 21:46:49.872854 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:46:49.870029    5053 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-serving-cert: secret "maas-api-e2e-ipp-a-3c0bdb-serving-cert" not found
Nov 03 21:46:49.872854 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:46:49.870102    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/986245b5-a0a6-4997-92aa-3b1da76ac041-maas-api-tls podName:986245b5-a0a6-4997-92aa-3b1da76ac041 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:50.370080995 +0000 UTC m=+1924.864002862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/986245b5-a0a6-4997-92aa-3b1da76ac041-maas-api-tls") pod "maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7" (UID: "986245b5-a0a6-4997-92aa-3b1da76ac041") : secret "maas-api-e2e-ipp-a-3c0bdb-serving-cert" not found
Nov 03 21:46:49.873693 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:49.873659    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/986245b5-a0a6-4997-92aa-3b1da76ac041-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7\" (UID: \"986245b5-a0a6-4997-92aa-3b1da76ac041\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"
Nov 03 21:46:49.878913 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:49.878732    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/986245b5-a0a6-4997-92aa-3b1da76ac041-metrics-certs\") pod \"maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7\" (UID: \"986245b5-a0a6-4997-92aa-3b1da76ac041\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"
Nov 03 21:46:49.930476 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:49.930446    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g2nx\" (UniqueName: \"kubernetes.io/projected/986245b5-a0a6-4997-92aa-3b1da76ac041-kube-api-access-8g2nx\") pod \"maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7\" (UID: \"986245b5-a0a6-4997-92aa-3b1da76ac041\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"
Nov 03 21:46:50.377695 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:50.377660    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/986245b5-a0a6-4997-92aa-3b1da76ac041-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7\" (UID: \"986245b5-a0a6-4997-92aa-3b1da76ac041\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"
Nov 03 21:46:50.380487 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:50.380457    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/986245b5-a0a6-4997-92aa-3b1da76ac041-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7\" (UID: \"986245b5-a0a6-4997-92aa-3b1da76ac041\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"
Nov 03 21:46:50.498316 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:50.498285    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"
Nov 03 21:46:50.880045 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:50.880015    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"]
Nov 03 21:46:50.881008 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:46:50.880979    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod986245b5_a0a6_4997_92aa_3b1da76ac041.slice/crio-1260ba2fd17613a475a9e29c7b08dc955aa58a11f6a9752fbf98446d81584b87 WatchSource:0}: Error finding container 1260ba2fd17613a475a9e29c7b08dc955aa58a11f6a9752fbf98446d81584b87: Status 404 returned error can't find the container with id 1260ba2fd17613a475a9e29c7b08dc955aa58a11f6a9752fbf98446d81584b87
Nov 03 21:46:50.976100 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:50.976058    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7" event={"ID":"986245b5-a0a6-4997-92aa-3b1da76ac041","Type":"ContainerStarted","Data":"1260ba2fd17613a475a9e29c7b08dc955aa58a11f6a9752fbf98446d81584b87"}
Nov 03 21:46:51.560536 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:51.560504    5053 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:46:51.560707 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:51.560557    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs" podUID="1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:46:51.984592 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:51.984557    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7" event={"ID":"986245b5-a0a6-4997-92aa-3b1da76ac041","Type":"ContainerStarted","Data":"16ebd88658d6ddaf541ca288f7d628ea6e671345672c23c1774aea6b62e39628"}
Nov 03 21:46:51.985050 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:51.984772    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"
Nov 03 21:46:52.148301 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:52.148212    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7" podStartSLOduration=2.363611202 podStartE2EDuration="3.148196925s" totalImagesPullingTime="784.585723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:52.146701858 +0000 UTC m=+1926.640623733" watchObservedRunningTime="2026-11-03 21:46:52.148196925 +0000 UTC m=+1926.642118798"
Nov 03 21:46:52.821753 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:52.821726    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:46:53.039261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.035862    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-token\" (UniqueName: \"kubernetes.io/projected/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-token\") pod \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") "
Nov 03 21:46:53.039261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.035967    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-credential-socket\") pod \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") "
Nov 03 21:46:53.039261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.036035    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istiod-ca-cert\") pod \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") "
Nov 03 21:46:53.039261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.036085    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-data\") pod \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") "
Nov 03 21:46:53.039261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.036122    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-podinfo\") pod \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") "
Nov 03 21:46:53.039261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.036171    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-workload-certs\") pod \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") "
Nov 03 21:46:53.039261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.036351    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-envoy\") pod \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") "
Nov 03 21:46:53.039261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.036417    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-kube-api-access-nbc98\" (UniqueName: \"kubernetes.io/projected/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-kube-api-access-nbc98\") pod \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") "
Nov 03 21:46:53.039261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.036463    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-workload-socket\") pod \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\" (UID: \"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344\") "
Nov 03 21:46:53.039261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.037393    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-workload-socket" pod "1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344" (UID: "1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:46:53.069717 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.044000    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istiod-ca-cert" pod "1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344" (UID: "1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:46:53.069717 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.044279    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-credential-socket" pod "1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344" (UID: "1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:46:53.069717 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.045779    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-workload-certs" pod "1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344" (UID: "1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:46:53.069717 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.045943    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-data" pod "1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344" (UID: "1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:46:53.069717 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.065039    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-podinfo" pod "1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344" (UID: "1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:46:53.072093 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.072033    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-kube-api-access-nbc98" pod "1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344" (UID: "1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344"). InnerVolumeSpecName "kube-api-access-nbc98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:46:53.077087 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.075439    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-envoy" pod "1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344" (UID: "1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:46:53.077985 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.077955    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-token" pod "1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344" (UID: "1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:46:53.079759 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.079728    5053 generic.go:356] "Generic (PLEG): container finished" podID="1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344" containerID="55cfe9759b5da932271ca0a43b7092fc4b245a3a3f5f68f60ac730d93a6c7985" exitCode=0
Nov 03 21:46:53.080799 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.080773    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"
Nov 03 21:46:53.084625 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.084591    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs" event={"ID":"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344","Type":"ContainerDied","Data":"55cfe9759b5da932271ca0a43b7092fc4b245a3a3f5f68f60ac730d93a6c7985"}
Nov 03 21:46:53.084751 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.084735    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs" event={"ID":"1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344","Type":"ContainerDied","Data":"87ca2dcddeef7d6bfc63c7637e49a11a2f52b82ae660598236a3fdf8f4f62556"}
Nov 03 21:46:53.084791 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.084773    5053 scope.go:122] "RemoveContainer" containerID="55cfe9759b5da932271ca0a43b7092fc4b245a3a3f5f68f60ac730d93a6c7985"
Nov 03 21:46:53.099370 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.099344    5053 scope.go:122] "RemoveContainer" containerID="55cfe9759b5da932271ca0a43b7092fc4b245a3a3f5f68f60ac730d93a6c7985"
Nov 03 21:46:53.099752 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:46:53.099728    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55cfe9759b5da932271ca0a43b7092fc4b245a3a3f5f68f60ac730d93a6c7985\": container with ID starting with 55cfe9759b5da932271ca0a43b7092fc4b245a3a3f5f68f60ac730d93a6c7985 not found: ID does not exist" containerID="55cfe9759b5da932271ca0a43b7092fc4b245a3a3f5f68f60ac730d93a6c7985"
Nov 03 21:46:53.099847 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.099764    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55cfe9759b5da932271ca0a43b7092fc4b245a3a3f5f68f60ac730d93a6c7985"} err="failed to get container status \"55cfe9759b5da932271ca0a43b7092fc4b245a3a3f5f68f60ac730d93a6c7985\": rpc error: code = NotFound desc = could not find container \"55cfe9759b5da932271ca0a43b7092fc4b245a3a3f5f68f60ac730d93a6c7985\": container with ID starting with 55cfe9759b5da932271ca0a43b7092fc4b245a3a3f5f68f60ac730d93a6c7985 not found: ID does not exist"
Nov 03 21:46:53.138580 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.138544    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:53.138580 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.138581    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:53.138918 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.138598    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:53.138918 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.138613    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:53.138918 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.138626    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:53.138918 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.138638    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:53.138918 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.138649    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:53.138918 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.138662    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:53.138918 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.138676    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbc98\" (UniqueName: \"kubernetes.io/projected/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344-kube-api-access-nbc98\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:53.250967 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.250930    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"]
Nov 03 21:46:53.254311 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.254275    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-8mnvs"]
Nov 03 21:46:53.690076 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:53.689508    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344" path="/var/lib/kubelet/pods/1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344/volumes"
Nov 03 21:46:58.092195 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:46:58.092162    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"
Nov 03 21:47:02.921702 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:02.920165    5053 patch_prober.go:29] interesting pod/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": read tcp 10.132.0.2:50054->10.132.0.174:15021: read: connection reset by peer" start-of-body=
Nov 03 21:47:02.921702 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:02.920268    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb" podUID="fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": read tcp 10.132.0.2:50054->10.132.0.174:15021: read: connection reset by peer"
Nov 03 21:47:03.177483 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:03.177381    5053 generic.go:356] "Generic (PLEG): container finished" podID="fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" containerID="009c614ee0742a684a3fa093be169b8d2fa2e56d364ae582be3c6742c4091432" exitCode=0
Nov 03 21:47:03.177483 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:03.177459    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb" event={"ID":"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc","Type":"ContainerDied","Data":"009c614ee0742a684a3fa093be169b8d2fa2e56d364ae582be3c6742c4091432"}
Nov 03 21:47:03.177977 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:03.177959    5053 scope.go:122] "RemoveContainer" containerID="009c614ee0742a684a3fa093be169b8d2fa2e56d364ae582be3c6742c4091432"
Nov 03 21:47:03.179488 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:03.179457    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:47:03.179589 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:03.179538    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:47:03.179660 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:03.179591    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:47:04.187392 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:04.187361    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb" event={"ID":"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc","Type":"ContainerStarted","Data":"3aa569c93d320d5324a27563bc7934d73745191b1a58e35d6f98fe31d8621595"}
Nov 03 21:47:04.219741 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:04.219711    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:47:04.219923 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:04.219871    5053 patch_prober.go:29] interesting pod/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Nov 03 21:47:04.219993 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:04.219934    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb" podUID="fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Nov 03 21:47:05.222705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:05.222664    5053 patch_prober.go:29] interesting pod/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Nov 03 21:47:05.223383 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:05.223351    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb" podUID="fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Nov 03 21:47:06.220261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:06.219653    5053 patch_prober.go:29] interesting pod/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Nov 03 21:47:06.220261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:06.219714    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb" podUID="fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Nov 03 21:47:07.219663 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:07.219627    5053 patch_prober.go:29] interesting pod/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Nov 03 21:47:07.220336 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:07.220302    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb" podUID="fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Nov 03 21:47:08.223261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:08.222686    5053 patch_prober.go:29] interesting pod/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Nov 03 21:47:08.223261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:08.222758    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb" podUID="fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Nov 03 21:47:09.219674 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:09.219635    5053 patch_prober.go:29] interesting pod/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Nov 03 21:47:09.220086 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:09.220053    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb" podUID="fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Nov 03 21:47:10.224543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:10.224503    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:47:10.225319 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:10.225288    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:47:10.226475 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:10.226452    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:47:11.009422 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:47:11.009389    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:47:21.424079 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:21.423974    5053 generic.go:356] "Generic (PLEG): container finished" podID="7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" containerID="80445fe3e688fc4de4fb5df38d984dc8c251ee1407fedc5c9e9bf675362a5a34" exitCode=0
Nov 03 21:47:21.424079 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:21.424051    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd" event={"ID":"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3","Type":"ContainerDied","Data":"80445fe3e688fc4de4fb5df38d984dc8c251ee1407fedc5c9e9bf675362a5a34"}
Nov 03 21:47:21.424776 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:21.424539    5053 scope.go:122] "RemoveContainer" containerID="80445fe3e688fc4de4fb5df38d984dc8c251ee1407fedc5c9e9bf675362a5a34"
Nov 03 21:47:21.426154 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:21.426120    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:47:21.426154 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:21.426145    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2" event={"ID":"1dc20893-5839-4b09-b930-9b9f150105fa","Type":"ContainerDied","Data":"35b65990e60f7449e382b9e2e3b20ffef53706f86b55874c33f94d6cd0dd9a51"}
Nov 03 21:47:21.426370 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:21.426212    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:47:21.426370 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:21.426266    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:47:21.426370 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:21.426127    5053 generic.go:356] "Generic (PLEG): container finished" podID="1dc20893-5839-4b09-b930-9b9f150105fa" containerID="35b65990e60f7449e382b9e2e3b20ffef53706f86b55874c33f94d6cd0dd9a51" exitCode=0
Nov 03 21:47:21.426558 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:21.426542    5053 scope.go:122] "RemoveContainer" containerID="35b65990e60f7449e382b9e2e3b20ffef53706f86b55874c33f94d6cd0dd9a51"
Nov 03 21:47:21.427896 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:21.427868    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:47:21.428008 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:21.427930    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:47:21.428008 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:21.427987    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:47:22.443701 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:22.443661    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd" event={"ID":"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3","Type":"ContainerStarted","Data":"0981d3f8d9298c3008c7367fb67fab6719327572a0555af81b81de3e2e30b01e"}
Nov 03 21:47:22.446299 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:22.446267    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2" event={"ID":"1dc20893-5839-4b09-b930-9b9f150105fa","Type":"ContainerStarted","Data":"2dc2caa1db2715c43516bb5ba922b388a2d9d99471b9a76f2eaf42b2c912b745"}
Nov 03 21:47:22.603891 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:22.603859    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:47:22.604279 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:22.604229    5053 patch_prober.go:29] interesting pod/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 03 21:47:22.604488 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:22.604464    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd" podUID="7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 03 21:47:23.000141 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:23.000061    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:47:23.001394 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:23.001366    5053 patch_prober.go:29] interesting pod/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 03 21:47:23.001596 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:23.001573    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2" podUID="1dc20893-5839-4b09-b930-9b9f150105fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 03 21:47:23.604588 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:23.604553    5053 patch_prober.go:29] interesting pod/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 03 21:47:23.605171 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:23.605139    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd" podUID="7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 03 21:47:24.000006 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:23.999973    5053 patch_prober.go:29] interesting pod/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 03 21:47:24.006266 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:24.000328    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2" podUID="1dc20893-5839-4b09-b930-9b9f150105fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 03 21:47:24.616262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:24.610641    5053 patch_prober.go:29] interesting pod/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 03 21:47:24.616262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:24.610702    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd" podUID="7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 03 21:47:25.004265 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:25.000222    5053 patch_prober.go:29] interesting pod/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 03 21:47:25.004265 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:25.000334    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2" podUID="1dc20893-5839-4b09-b930-9b9f150105fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 03 21:47:25.404520 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:25.404402    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:47:25.409258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:25.408330    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:47:25.604265 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:25.603573    5053 patch_prober.go:29] interesting pod/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 03 21:47:25.604265 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:25.603632    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd" podUID="7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 03 21:47:25.999769 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:25.999737    5053 patch_prober.go:29] interesting pod/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 03 21:47:26.000196 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:25.999791    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2" podUID="1dc20893-5839-4b09-b930-9b9f150105fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 03 21:47:26.619263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:26.605955    5053 patch_prober.go:29] interesting pod/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 03 21:47:26.619263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:26.606014    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd" podUID="7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 03 21:47:27.011768 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:27.011723    5053 patch_prober.go:29] interesting pod/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 03 21:47:27.012361 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:27.011787    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2" podUID="1dc20893-5839-4b09-b930-9b9f150105fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 03 21:47:27.611100 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:27.610627    5053 patch_prober.go:29] interesting pod/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 03 21:47:27.611100 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:27.610710    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd" podUID="7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 03 21:47:28.166266 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:28.165568    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:47:28.169265 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:28.167842    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:47:28.608066 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:28.608036    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:47:28.609216 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:28.609189    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:47:40.303901 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:40.303865    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"]
Nov 03 21:47:40.305921 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:40.305890    5053 cpu_manager.go:414] "containerMap: removing container" podUID="1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344" containerName="istio-proxy"
Nov 03 21:47:40.306102 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:40.306089    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344" containerName="istio-proxy"
Nov 03 21:47:40.306513 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:40.306492    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cf2bd9c-1e8e-41eb-b19f-bf150ee8e344" containerName="istio-proxy"
Nov 03 21:47:40.311887 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:40.311863    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"
Nov 03 21:47:40.353473 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:40.353432    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9wqh\" (UniqueName: \"kubernetes.io/projected/84aa8472-f94e-4195-94e8-a83e9bc596d8-kube-api-access-m9wqh\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c\" (UID: \"84aa8472-f94e-4195-94e8-a83e9bc596d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"
Nov 03 21:47:40.353663 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:40.353580    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84aa8472-f94e-4195-94e8-a83e9bc596d8-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c\" (UID: \"84aa8472-f94e-4195-94e8-a83e9bc596d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"
Nov 03 21:47:40.353663 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:40.353642    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84aa8472-f94e-4195-94e8-a83e9bc596d8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c\" (UID: \"84aa8472-f94e-4195-94e8-a83e9bc596d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"
Nov 03 21:47:40.353831 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:40.353805    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84aa8472-f94e-4195-94e8-a83e9bc596d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c\" (UID: \"84aa8472-f94e-4195-94e8-a83e9bc596d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"
Nov 03 21:47:40.361270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:40.361200    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"]
Nov 03 21:47:40.406402 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:40.406365    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-6ddbba-serving-cert\""
Nov 03 21:47:40.455174 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:40.455126    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84aa8472-f94e-4195-94e8-a83e9bc596d8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c\" (UID: \"84aa8472-f94e-4195-94e8-a83e9bc596d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"
Nov 03 21:47:40.455399 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:40.455340    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84aa8472-f94e-4195-94e8-a83e9bc596d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c\" (UID: \"84aa8472-f94e-4195-94e8-a83e9bc596d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"
Nov 03 21:47:40.455399 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:40.455388    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9wqh\" (UniqueName: \"kubernetes.io/projected/84aa8472-f94e-4195-94e8-a83e9bc596d8-kube-api-access-m9wqh\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c\" (UID: \"84aa8472-f94e-4195-94e8-a83e9bc596d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"
Nov 03 21:47:40.455545 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:40.455511    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84aa8472-f94e-4195-94e8-a83e9bc596d8-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c\" (UID: \"84aa8472-f94e-4195-94e8-a83e9bc596d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"
Nov 03 21:47:40.456567 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:47:40.456486    5053 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-serving-cert: secret "maas-api-e2e-shared-b-w2-6ddbba-serving-cert" not found
Nov 03 21:47:40.456567 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:47:40.456567    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84aa8472-f94e-4195-94e8-a83e9bc596d8-maas-api-tls podName:84aa8472-f94e-4195-94e8-a83e9bc596d8 nodeName:}" failed. No retries permitted until 2026-11-03 21:47:40.956547504 +0000 UTC m=+1975.450469380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/84aa8472-f94e-4195-94e8-a83e9bc596d8-maas-api-tls") pod "maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c" (UID: "84aa8472-f94e-4195-94e8-a83e9bc596d8") : secret "maas-api-e2e-shared-b-w2-6ddbba-serving-cert" not found
Nov 03 21:47:40.456774 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:40.456732    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84aa8472-f94e-4195-94e8-a83e9bc596d8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c\" (UID: \"84aa8472-f94e-4195-94e8-a83e9bc596d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"
Nov 03 21:47:40.458637 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:40.458616    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84aa8472-f94e-4195-94e8-a83e9bc596d8-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c\" (UID: \"84aa8472-f94e-4195-94e8-a83e9bc596d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"
Nov 03 21:47:40.514055 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:40.514026    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9wqh\" (UniqueName: \"kubernetes.io/projected/84aa8472-f94e-4195-94e8-a83e9bc596d8-kube-api-access-m9wqh\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c\" (UID: \"84aa8472-f94e-4195-94e8-a83e9bc596d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"
Nov 03 21:47:40.929614 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:47:40.929577    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:47:40.966020 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:40.965981    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84aa8472-f94e-4195-94e8-a83e9bc596d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c\" (UID: \"84aa8472-f94e-4195-94e8-a83e9bc596d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"
Nov 03 21:47:40.968551 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:40.968529    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84aa8472-f94e-4195-94e8-a83e9bc596d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c\" (UID: \"84aa8472-f94e-4195-94e8-a83e9bc596d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"
Nov 03 21:47:41.227222 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:41.227184    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"
Nov 03 21:47:41.610653 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:41.610620    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"]
Nov 03 21:47:41.611736 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:47:41.611705    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84aa8472_f94e_4195_94e8_a83e9bc596d8.slice/crio-3815cb140e88fcbe9cd332555225b2b2e096fef8450f15e323957a4578d628e9 WatchSource:0}: Error finding container 3815cb140e88fcbe9cd332555225b2b2e096fef8450f15e323957a4578d628e9: Status 404 returned error can't find the container with id 3815cb140e88fcbe9cd332555225b2b2e096fef8450f15e323957a4578d628e9
Nov 03 21:47:41.613181 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:41.613165    5053 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:47:41.694946 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:41.694913    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c" event={"ID":"84aa8472-f94e-4195-94e8-a83e9bc596d8","Type":"ContainerStarted","Data":"3815cb140e88fcbe9cd332555225b2b2e096fef8450f15e323957a4578d628e9"}
Nov 03 21:47:42.703988 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:42.703950    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c" event={"ID":"84aa8472-f94e-4195-94e8-a83e9bc596d8","Type":"ContainerStarted","Data":"d9f5651d1e96fa32861019181a23eb33286ad9ac03e6ba69fe3ac862d2790e5d"}
Nov 03 21:47:42.704473 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:42.704019    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"
Nov 03 21:47:42.821610 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:42.821548    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c" podStartSLOduration=1.9982437210000001 podStartE2EDuration="2.821531955s" totalImagesPullingTime="823.288234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:47:42.818097423 +0000 UTC m=+1977.312019299" watchObservedRunningTime="2026-11-03 21:47:42.821531955 +0000 UTC m=+1977.315453833"
Nov 03 21:47:47.192201 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:47.192164    5053 scope.go:122] "RemoveContainer" containerID="1ebdeabedab65cccbaaa91cf46fc8090368e1634325addb6c8196d0a733da5d6"
Nov 03 21:47:48.715680 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:47:48.715647    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"
Nov 03 21:48:10.938902 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:48:10.938822    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:48:13.978430 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:13.978388    5053 generic.go:356] "Generic (PLEG): container finished" podID="d17d1dfe-8556-43d4-b67f-d0e628552f0b" containerID="adc7e544a6d458cdd2ef0b533d07f79cdca60a608588c54144966b1b578bed41" exitCode=0
Nov 03 21:48:13.978894 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:13.978456    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs" event={"ID":"d17d1dfe-8556-43d4-b67f-d0e628552f0b","Type":"ContainerDied","Data":"adc7e544a6d458cdd2ef0b533d07f79cdca60a608588c54144966b1b578bed41"}
Nov 03 21:48:13.978894 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:13.978889    5053 scope.go:122] "RemoveContainer" containerID="adc7e544a6d458cdd2ef0b533d07f79cdca60a608588c54144966b1b578bed41"
Nov 03 21:48:13.980469 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:13.980436    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:48:13.980575 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:13.980507    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:48:13.980575 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:13.980539    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:48:14.987285 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:14.987232    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs" event={"ID":"d17d1dfe-8556-43d4-b67f-d0e628552f0b","Type":"ContainerStarted","Data":"cf529980f5c953275d110a322ade4ef906f85992c34846c724e61fa8ba713068"}
Nov 03 21:48:15.394232 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:15.394150    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:48:15.394409 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:15.394389    5053 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 03 21:48:15.394454 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:15.394433    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs" podUID="d17d1dfe-8556-43d4-b67f-d0e628552f0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 03 21:48:16.394985 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:16.394949    5053 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 03 21:48:16.395440 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:16.395002    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs" podUID="d17d1dfe-8556-43d4-b67f-d0e628552f0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 03 21:48:17.395046 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:17.395009    5053 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 03 21:48:17.395474 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:17.395065    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs" podUID="d17d1dfe-8556-43d4-b67f-d0e628552f0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 03 21:48:18.394071 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:18.394037    5053 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 03 21:48:18.394373 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:18.394089    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs" podUID="d17d1dfe-8556-43d4-b67f-d0e628552f0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 03 21:48:19.394548 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:19.394514    5053 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 03 21:48:19.394983 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:19.394591    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs" podUID="d17d1dfe-8556-43d4-b67f-d0e628552f0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 03 21:48:20.399358 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:20.399328    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:48:20.399796 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:20.399768    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:48:20.400394 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:20.400373    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 21:48:40.961516 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:48:40.961477    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:48:47.223286 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:47.223254    5053 scope.go:122] "RemoveContainer" containerID="a76183f97b79d58f9c0dd4b91701b3a54f85fefdc99e5254c753a073b54377e0"
Nov 03 21:48:47.233824 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:47.233804    5053 scope.go:122] "RemoveContainer" containerID="c75df8d88a30595c26b9358a9c195cf96704484d5a6643d43d992a5610986b1b"
Nov 03 21:48:47.243485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:47.243465    5053 scope.go:122] "RemoveContainer" containerID="e7df406340219f4287ba7dd8ef0fad0237c20f38052f0bfad894ddf7dd3e886a"
Nov 03 21:48:47.253171 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:48:47.253148    5053 scope.go:122] "RemoveContainer" containerID="dc42239d89451d040ec7b83e7c250232e94d3cad85eee8ad3a86b75199eef926"
Nov 03 21:49:10.930121 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:49:10.930089    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:49:13.906352 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:13.906318    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"]
Nov 03 21:49:13.910830 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:13.910810    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"
Nov 03 21:49:13.963947 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:13.963919    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-62cf5c-serving-cert\""
Nov 03 21:49:13.979222 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:13.979183    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e6dbdf0-e267-42b5-a916-4f39502de63d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9\" (UID: \"1e6dbdf0-e267-42b5-a916-4f39502de63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"
Nov 03 21:49:13.979441 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:13.979368    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e6dbdf0-e267-42b5-a916-4f39502de63d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9\" (UID: \"1e6dbdf0-e267-42b5-a916-4f39502de63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"
Nov 03 21:49:13.979441 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:13.979407    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e6dbdf0-e267-42b5-a916-4f39502de63d-metrics-certs\") pod \"maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9\" (UID: \"1e6dbdf0-e267-42b5-a916-4f39502de63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"
Nov 03 21:49:13.979441 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:13.979439    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnrmt\" (UniqueName: \"kubernetes.io/projected/1e6dbdf0-e267-42b5-a916-4f39502de63d-kube-api-access-tnrmt\") pod \"maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9\" (UID: \"1e6dbdf0-e267-42b5-a916-4f39502de63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"
Nov 03 21:49:14.017345 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:14.017312    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"]
Nov 03 21:49:14.080902 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:14.080870    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e6dbdf0-e267-42b5-a916-4f39502de63d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9\" (UID: \"1e6dbdf0-e267-42b5-a916-4f39502de63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"
Nov 03 21:49:14.081165 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:14.080912    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e6dbdf0-e267-42b5-a916-4f39502de63d-metrics-certs\") pod \"maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9\" (UID: \"1e6dbdf0-e267-42b5-a916-4f39502de63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"
Nov 03 21:49:14.081165 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:14.080946    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnrmt\" (UniqueName: \"kubernetes.io/projected/1e6dbdf0-e267-42b5-a916-4f39502de63d-kube-api-access-tnrmt\") pod \"maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9\" (UID: \"1e6dbdf0-e267-42b5-a916-4f39502de63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"
Nov 03 21:49:14.081165 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:14.081081    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e6dbdf0-e267-42b5-a916-4f39502de63d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9\" (UID: \"1e6dbdf0-e267-42b5-a916-4f39502de63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"
Nov 03 21:49:14.081399 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:49:14.081373    5053 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-serving-cert: secret "maas-api-e2e-ipp-b-62cf5c-serving-cert" not found
Nov 03 21:49:14.081462 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:49:14.081437    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e6dbdf0-e267-42b5-a916-4f39502de63d-maas-api-tls podName:1e6dbdf0-e267-42b5-a916-4f39502de63d nodeName:}" failed. No retries permitted until 2026-11-03 21:49:14.581418061 +0000 UTC m=+2069.075339920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1e6dbdf0-e267-42b5-a916-4f39502de63d-maas-api-tls") pod "maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9" (UID: "1e6dbdf0-e267-42b5-a916-4f39502de63d") : secret "maas-api-e2e-ipp-b-62cf5c-serving-cert" not found
Nov 03 21:49:14.081921 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:14.081861    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e6dbdf0-e267-42b5-a916-4f39502de63d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9\" (UID: \"1e6dbdf0-e267-42b5-a916-4f39502de63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"
Nov 03 21:49:14.083879 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:14.083858    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e6dbdf0-e267-42b5-a916-4f39502de63d-metrics-certs\") pod \"maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9\" (UID: \"1e6dbdf0-e267-42b5-a916-4f39502de63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"
Nov 03 21:49:14.138605 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:14.138568    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnrmt\" (UniqueName: \"kubernetes.io/projected/1e6dbdf0-e267-42b5-a916-4f39502de63d-kube-api-access-tnrmt\") pod \"maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9\" (UID: \"1e6dbdf0-e267-42b5-a916-4f39502de63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"
Nov 03 21:49:14.588481 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:14.588439    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e6dbdf0-e267-42b5-a916-4f39502de63d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9\" (UID: \"1e6dbdf0-e267-42b5-a916-4f39502de63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"
Nov 03 21:49:14.591197 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:14.591169    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e6dbdf0-e267-42b5-a916-4f39502de63d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9\" (UID: \"1e6dbdf0-e267-42b5-a916-4f39502de63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"
Nov 03 21:49:14.824956 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:14.824916    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"
Nov 03 21:49:15.205052 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:15.205006    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"]
Nov 03 21:49:15.205705 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:49:15.205673    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6dbdf0_e267_42b5_a916_4f39502de63d.slice/crio-25b9d4233e43050cd482c320c18cd864789b5739056730a9aecedf1601208343 WatchSource:0}: Error finding container 25b9d4233e43050cd482c320c18cd864789b5739056730a9aecedf1601208343: Status 404 returned error can't find the container with id 25b9d4233e43050cd482c320c18cd864789b5739056730a9aecedf1601208343
Nov 03 21:49:15.533377 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:15.533295    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9" event={"ID":"1e6dbdf0-e267-42b5-a916-4f39502de63d","Type":"ContainerStarted","Data":"25b9d4233e43050cd482c320c18cd864789b5739056730a9aecedf1601208343"}
Nov 03 21:49:16.543676 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:16.543643    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9" event={"ID":"1e6dbdf0-e267-42b5-a916-4f39502de63d","Type":"ContainerStarted","Data":"b82a4debb44d1ef7340e5b4fafd93a7999472fb0bf1c36f04a2b0110d8aca917"}
Nov 03 21:49:16.544095 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:16.543736    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"
Nov 03 21:49:16.659799 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:16.659744    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9" podStartSLOduration=2.842874684 podStartE2EDuration="3.659731027s" totalImagesPullingTime="816.856343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:16.656626859 +0000 UTC m=+2071.150548734" watchObservedRunningTime="2026-11-03 21:49:16.659731027 +0000 UTC m=+2071.153652901"
Nov 03 21:49:22.558542 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:22.558507    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"
Nov 03 21:49:40.928590 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:49:40.928554    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:49:45.997915 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:45.997874    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 21:49:46.006466 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:46.006443    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:49:46.010841 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:46.010823    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 21:49:46.018536 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:46.018519    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:49:47.308870 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:47.308837    5053 scope.go:122] "RemoveContainer" containerID="b639292af1a02a12d4f09ca3a8408cdc737a73c6ae0621c068b2cf140cfee4e6"
Nov 03 21:49:48.050210 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:48.050182    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b9ed93-4rztg"]
Nov 03 21:49:48.099472 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:48.099443    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b9ed93-4rztg"]
Nov 03 21:49:49.668929 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:49:49.668895    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c61761e5-ce80-491c-b6f0-ed22df43ad19" path="/var/lib/kubelet/pods/c61761e5-ce80-491c-b6f0-ed22df43ad19/volumes"
Nov 03 21:50:10.929031 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:50:10.928993    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:50:40.928753 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:50:40.928717    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:50:47.335641 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:50:47.335608    5053 scope.go:122] "RemoveContainer" containerID="6bb82cb57fea8944136fa9288435fe7c05eef3cdb53f10074ea8aa38a9fdc1c2"
Nov 03 21:50:47.347948 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:50:47.347927    5053 scope.go:122] "RemoveContainer" containerID="74fb0c452a2d30e92c789dbd05861ed72cda0a0fc766f8c141fc8383d839b938"
Nov 03 21:50:47.358060 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:50:47.358037    5053 scope.go:122] "RemoveContainer" containerID="4d835e2cfa89e43c4073100db85d2b585533efd51a2dbf0229e58bdb41c37792"
Nov 03 21:51:07.226167 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:07.226135    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"]
Nov 03 21:51:07.226580 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:07.226388    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg" podUID="5d7bc971-f569-4f7c-88bf-5e3168bd49bc" containerName="istio-proxy" containerID="cri-o://a7f3e462d4e398b3d733d20503cd3b342c84a2df3801b5f2782abe1ffc4cfde3" gracePeriod=30
Nov 03 21:51:10.991563 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:51:10.991493    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:51:12.369121 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.369076    5053 generic.go:356] "Generic (PLEG): container finished" podID="5d7bc971-f569-4f7c-88bf-5e3168bd49bc" containerID="a7f3e462d4e398b3d733d20503cd3b342c84a2df3801b5f2782abe1ffc4cfde3" exitCode=0
Nov 03 21:51:12.369547 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.369145    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg" event={"ID":"5d7bc971-f569-4f7c-88bf-5e3168bd49bc","Type":"ContainerDied","Data":"a7f3e462d4e398b3d733d20503cd3b342c84a2df3801b5f2782abe1ffc4cfde3"}
Nov 03 21:51:12.534229 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.534203    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:51:12.682530 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.682496    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-workload-certs\") pod \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") "
Nov 03 21:51:12.682728 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.682557    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-credential-socket\") pod \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") "
Nov 03 21:51:12.682728 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.682598    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-data\") pod \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") "
Nov 03 21:51:12.682728 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.682680    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-token\" (UniqueName: \"kubernetes.io/projected/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-token\") pod \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") "
Nov 03 21:51:12.682919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.682750    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-workload-socket\") pod \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") "
Nov 03 21:51:12.682919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.682786    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-envoy\") pod \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") "
Nov 03 21:51:12.682919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.682830    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-podinfo\") pod \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") "
Nov 03 21:51:12.682919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.682855    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-workload-certs" pod "5d7bc971-f569-4f7c-88bf-5e3168bd49bc" (UID: "5d7bc971-f569-4f7c-88bf-5e3168bd49bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:12.682919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.682892    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-kube-api-access-dcjbd\" (UniqueName: \"kubernetes.io/projected/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-kube-api-access-dcjbd\") pod \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") "
Nov 03 21:51:12.683213 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.682919    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-credential-socket" pod "5d7bc971-f569-4f7c-88bf-5e3168bd49bc" (UID: "5d7bc971-f569-4f7c-88bf-5e3168bd49bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:12.683213 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.682964    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istiod-ca-cert\") pod \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\" (UID: \"5d7bc971-f569-4f7c-88bf-5e3168bd49bc\") "
Nov 03 21:51:12.683384 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.683359    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-workload-socket" pod "5d7bc971-f569-4f7c-88bf-5e3168bd49bc" (UID: "5d7bc971-f569-4f7c-88bf-5e3168bd49bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:12.683656 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.683634    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-data" pod "5d7bc971-f569-4f7c-88bf-5e3168bd49bc" (UID: "5d7bc971-f569-4f7c-88bf-5e3168bd49bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:12.683786 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.683650    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istiod-ca-cert" pod "5d7bc971-f569-4f7c-88bf-5e3168bd49bc" (UID: "5d7bc971-f569-4f7c-88bf-5e3168bd49bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:51:12.684111 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.683644    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:12.684260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.684126    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:12.684260 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.684146    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:12.685823 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.685801    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-token" pod "5d7bc971-f569-4f7c-88bf-5e3168bd49bc" (UID: "5d7bc971-f569-4f7c-88bf-5e3168bd49bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:51:12.686277 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.686214    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-podinfo" pod "5d7bc971-f569-4f7c-88bf-5e3168bd49bc" (UID: "5d7bc971-f569-4f7c-88bf-5e3168bd49bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:51:12.686277 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.686225    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-envoy" pod "5d7bc971-f569-4f7c-88bf-5e3168bd49bc" (UID: "5d7bc971-f569-4f7c-88bf-5e3168bd49bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:12.687338 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.687317    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-kube-api-access-dcjbd" pod "5d7bc971-f569-4f7c-88bf-5e3168bd49bc" (UID: "5d7bc971-f569-4f7c-88bf-5e3168bd49bc"). InnerVolumeSpecName "kube-api-access-dcjbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:51:12.785567 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.785483    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:12.785567 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.785514    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:12.785567 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.785525    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:12.785567 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.785533    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcjbd\" (UniqueName: \"kubernetes.io/projected/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-kube-api-access-dcjbd\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:12.785567 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.785543    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:12.785567 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:12.785550    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d7bc971-f569-4f7c-88bf-5e3168bd49bc-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:13.378310 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:13.378278    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"
Nov 03 21:51:13.378310 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:13.378291    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg" event={"ID":"5d7bc971-f569-4f7c-88bf-5e3168bd49bc","Type":"ContainerDied","Data":"3d6f6062457a2a033fa8e73b602d822be2ce3f663b476f8564afb645ac2df8d1"}
Nov 03 21:51:13.378874 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:13.378346    5053 scope.go:122] "RemoveContainer" containerID="a7f3e462d4e398b3d733d20503cd3b342c84a2df3801b5f2782abe1ffc4cfde3"
Nov 03 21:51:13.592949 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:13.592910    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"]
Nov 03 21:51:13.644494 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:13.644414    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4s8qg"]
Nov 03 21:51:13.667208 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:13.667174    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d7bc971-f569-4f7c-88bf-5e3168bd49bc" path="/var/lib/kubelet/pods/5d7bc971-f569-4f7c-88bf-5e3168bd49bc/volumes"
Nov 03 21:51:13.901991 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:13.901790    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"]
Nov 03 21:51:13.902789 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:13.902736    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96" podUID="a570d7ee-ec67-4db1-a2d2-0ffc18b0d439" containerName="istio-proxy" containerID="cri-o://b1789c94394ea93bdcee8f0efdee483e716e93e3a82edc31ab4d72826b329685" gracePeriod=30
Nov 03 21:51:18.936640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:18.936600    5053 patch_prober.go:29] interesting pod/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:51:18.937069 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:18.936680    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96" podUID="a570d7ee-ec67-4db1-a2d2-0ffc18b0d439" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:51:18.952892 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:18.952864    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"]
Nov 03 21:51:18.953135 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:18.953093    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6" podUID="7f68c476-0110-4dd1-a322-e7f0e991f56b" containerName="discovery" containerID="cri-o://2413071844b82bf1f0a0c36fc99c76cbfd94778e0ca1cdddfc3c042e70bf3117" gracePeriod=30
Nov 03 21:51:19.272230 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.272203    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:51:19.292541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.292517    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:51:19.296626 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.296605    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-kube-api-access-m6p7w\" (UniqueName: \"kubernetes.io/projected/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-kube-api-access-m6p7w\") pod \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") "
Nov 03 21:51:19.296702 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.296654    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istiod-ca-cert\") pod \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") "
Nov 03 21:51:19.296756 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.296700    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-workload-certs\") pod \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") "
Nov 03 21:51:19.296810 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.296772    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-token\" (UniqueName: \"kubernetes.io/projected/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-token\") pod \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") "
Nov 03 21:51:19.296810 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.296790    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-credential-socket\") pod \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") "
Nov 03 21:51:19.296916 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.296825    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-envoy\") pod \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") "
Nov 03 21:51:19.296916 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.296904    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-workload-socket\") pod \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") "
Nov 03 21:51:19.297048 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.297011    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-podinfo\") pod \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") "
Nov 03 21:51:19.297113 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.297057    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-data\") pod \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\" (UID: \"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439\") "
Nov 03 21:51:19.297113 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.297064    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-workload-certs" pod "a570d7ee-ec67-4db1-a2d2-0ffc18b0d439" (UID: "a570d7ee-ec67-4db1-a2d2-0ffc18b0d439"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:19.297113 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.297092    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-credential-socket" pod "a570d7ee-ec67-4db1-a2d2-0ffc18b0d439" (UID: "a570d7ee-ec67-4db1-a2d2-0ffc18b0d439"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:19.297300 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.297142    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istiod-ca-cert" pod "a570d7ee-ec67-4db1-a2d2-0ffc18b0d439" (UID: "a570d7ee-ec67-4db1-a2d2-0ffc18b0d439"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:51:19.297488 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.297464    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-workload-socket" pod "a570d7ee-ec67-4db1-a2d2-0ffc18b0d439" (UID: "a570d7ee-ec67-4db1-a2d2-0ffc18b0d439"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:19.297602 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.297505    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-data" pod "a570d7ee-ec67-4db1-a2d2-0ffc18b0d439" (UID: "a570d7ee-ec67-4db1-a2d2-0ffc18b0d439"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:19.297915 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.297786    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:19.297971 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.297921    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:19.297971 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.297937    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:19.297971 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.297950    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:19.297971 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.297963    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:19.299130 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.299107    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-envoy" pod "a570d7ee-ec67-4db1-a2d2-0ffc18b0d439" (UID: "a570d7ee-ec67-4db1-a2d2-0ffc18b0d439"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:19.299294 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.299275    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-podinfo" pod "a570d7ee-ec67-4db1-a2d2-0ffc18b0d439" (UID: "a570d7ee-ec67-4db1-a2d2-0ffc18b0d439"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:51:19.299572 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.299541    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-token" pod "a570d7ee-ec67-4db1-a2d2-0ffc18b0d439" (UID: "a570d7ee-ec67-4db1-a2d2-0ffc18b0d439"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:51:19.299572 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.299553    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-kube-api-access-m6p7w" pod "a570d7ee-ec67-4db1-a2d2-0ffc18b0d439" (UID: "a570d7ee-ec67-4db1-a2d2-0ffc18b0d439"). InnerVolumeSpecName "kube-api-access-m6p7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:51:19.398560 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.398527    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f68c476-0110-4dd1-a322-e7f0e991f56b-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f68c476-0110-4dd1-a322-e7f0e991f56b-local-certs\") pod \"7f68c476-0110-4dd1-a322-e7f0e991f56b\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") "
Nov 03 21:51:19.398747 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.398604    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-csr-dns-cert\") pod \"7f68c476-0110-4dd1-a322-e7f0e991f56b\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") "
Nov 03 21:51:19.398747 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.398706    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f68c476-0110-4dd1-a322-e7f0e991f56b-kube-api-access-dnw6g\" (UniqueName: \"kubernetes.io/projected/7f68c476-0110-4dd1-a322-e7f0e991f56b-kube-api-access-dnw6g\") pod \"7f68c476-0110-4dd1-a322-e7f0e991f56b\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") "
Nov 03 21:51:19.398876 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.398758    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-kubeconfig\") pod \"7f68c476-0110-4dd1-a322-e7f0e991f56b\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") "
Nov 03 21:51:19.398876 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.398799    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f68c476-0110-4dd1-a322-e7f0e991f56b-cacerts\" (UniqueName: \"kubernetes.io/secret/7f68c476-0110-4dd1-a322-e7f0e991f56b-cacerts\") pod \"7f68c476-0110-4dd1-a322-e7f0e991f56b\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") "
Nov 03 21:51:19.398982 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.398933    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-csr-ca-configmap\") pod \"7f68c476-0110-4dd1-a322-e7f0e991f56b\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") "
Nov 03 21:51:19.398982 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.398960    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-token\" (UniqueName: \"kubernetes.io/projected/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-token\") pod \"7f68c476-0110-4dd1-a322-e7f0e991f56b\" (UID: \"7f68c476-0110-4dd1-a322-e7f0e991f56b\") "
Nov 03 21:51:19.399855 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.399823    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6p7w\" (UniqueName: \"kubernetes.io/projected/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-kube-api-access-m6p7w\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:19.400000 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.399860    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:19.400000 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.399877    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:19.400000 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.399892    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:19.402182 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.402154    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f68c476-0110-4dd1-a322-e7f0e991f56b-local-certs" pod "7f68c476-0110-4dd1-a322-e7f0e991f56b" (UID: "7f68c476-0110-4dd1-a322-e7f0e991f56b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:19.402383 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.402357    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-kubeconfig" pod "7f68c476-0110-4dd1-a322-e7f0e991f56b" (UID: "7f68c476-0110-4dd1-a322-e7f0e991f56b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:51:19.402497 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.402478    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-csr-ca-configmap" pod "7f68c476-0110-4dd1-a322-e7f0e991f56b" (UID: "7f68c476-0110-4dd1-a322-e7f0e991f56b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:51:19.413807 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.413769    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-token" pod "7f68c476-0110-4dd1-a322-e7f0e991f56b" (UID: "7f68c476-0110-4dd1-a322-e7f0e991f56b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:51:19.413987 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.413958    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-csr-dns-cert" pod "7f68c476-0110-4dd1-a322-e7f0e991f56b" (UID: "7f68c476-0110-4dd1-a322-e7f0e991f56b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:51:19.414400 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.414375    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f68c476-0110-4dd1-a322-e7f0e991f56b-kube-api-access-dnw6g" pod "7f68c476-0110-4dd1-a322-e7f0e991f56b" (UID: "7f68c476-0110-4dd1-a322-e7f0e991f56b"). InnerVolumeSpecName "kube-api-access-dnw6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:51:19.415396 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.415343    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f68c476-0110-4dd1-a322-e7f0e991f56b-cacerts" pod "7f68c476-0110-4dd1-a322-e7f0e991f56b" (UID: "7f68c476-0110-4dd1-a322-e7f0e991f56b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:51:19.426402 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.426372    5053 generic.go:356] "Generic (PLEG): container finished" podID="a570d7ee-ec67-4db1-a2d2-0ffc18b0d439" containerID="b1789c94394ea93bdcee8f0efdee483e716e93e3a82edc31ab4d72826b329685" exitCode=0
Nov 03 21:51:19.426524 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.426438    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"
Nov 03 21:51:19.426588 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.426546    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96" event={"ID":"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439","Type":"ContainerDied","Data":"b1789c94394ea93bdcee8f0efdee483e716e93e3a82edc31ab4d72826b329685"}
Nov 03 21:51:19.426638 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.426593    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96" event={"ID":"a570d7ee-ec67-4db1-a2d2-0ffc18b0d439","Type":"ContainerDied","Data":"b1c7624a79882f78e057ec9ce5ffe66fef7f60103e8c41c5c3c0bf95c59b060a"}
Nov 03 21:51:19.426638 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.426620    5053 scope.go:122] "RemoveContainer" containerID="b1789c94394ea93bdcee8f0efdee483e716e93e3a82edc31ab4d72826b329685"
Nov 03 21:51:19.430517 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.429111    5053 generic.go:356] "Generic (PLEG): container finished" podID="7f68c476-0110-4dd1-a322-e7f0e991f56b" containerID="2413071844b82bf1f0a0c36fc99c76cbfd94778e0ca1cdddfc3c042e70bf3117" exitCode=0
Nov 03 21:51:19.430517 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.429177    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6" event={"ID":"7f68c476-0110-4dd1-a322-e7f0e991f56b","Type":"ContainerDied","Data":"2413071844b82bf1f0a0c36fc99c76cbfd94778e0ca1cdddfc3c042e70bf3117"}
Nov 03 21:51:19.430517 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.429213    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6" event={"ID":"7f68c476-0110-4dd1-a322-e7f0e991f56b","Type":"ContainerDied","Data":"b96a28d0e7eb54b60fac9f08458806e68ef66a817cd116d0f11e9007a56d41d0"}
Nov 03 21:51:19.430953 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.430816    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"
Nov 03 21:51:19.449447 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.446392    5053 scope.go:122] "RemoveContainer" containerID="b1789c94394ea93bdcee8f0efdee483e716e93e3a82edc31ab4d72826b329685"
Nov 03 21:51:19.449447 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:51:19.446744    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1789c94394ea93bdcee8f0efdee483e716e93e3a82edc31ab4d72826b329685\": container with ID starting with b1789c94394ea93bdcee8f0efdee483e716e93e3a82edc31ab4d72826b329685 not found: ID does not exist" containerID="b1789c94394ea93bdcee8f0efdee483e716e93e3a82edc31ab4d72826b329685"
Nov 03 21:51:19.449447 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.446772    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1789c94394ea93bdcee8f0efdee483e716e93e3a82edc31ab4d72826b329685"} err="failed to get container status \"b1789c94394ea93bdcee8f0efdee483e716e93e3a82edc31ab4d72826b329685\": rpc error: code = NotFound desc = could not find container \"b1789c94394ea93bdcee8f0efdee483e716e93e3a82edc31ab4d72826b329685\": container with ID starting with b1789c94394ea93bdcee8f0efdee483e716e93e3a82edc31ab4d72826b329685 not found: ID does not exist"
Nov 03 21:51:19.449447 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.446789    5053 scope.go:122] "RemoveContainer" containerID="2413071844b82bf1f0a0c36fc99c76cbfd94778e0ca1cdddfc3c042e70bf3117"
Nov 03 21:51:19.458459 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.458423    5053 scope.go:122] "RemoveContainer" containerID="2413071844b82bf1f0a0c36fc99c76cbfd94778e0ca1cdddfc3c042e70bf3117"
Nov 03 21:51:19.458760 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:51:19.458740    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2413071844b82bf1f0a0c36fc99c76cbfd94778e0ca1cdddfc3c042e70bf3117\": container with ID starting with 2413071844b82bf1f0a0c36fc99c76cbfd94778e0ca1cdddfc3c042e70bf3117 not found: ID does not exist" containerID="2413071844b82bf1f0a0c36fc99c76cbfd94778e0ca1cdddfc3c042e70bf3117"
Nov 03 21:51:19.458817 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.458767    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2413071844b82bf1f0a0c36fc99c76cbfd94778e0ca1cdddfc3c042e70bf3117"} err="failed to get container status \"2413071844b82bf1f0a0c36fc99c76cbfd94778e0ca1cdddfc3c042e70bf3117\": rpc error: code = NotFound desc = could not find container \"2413071844b82bf1f0a0c36fc99c76cbfd94778e0ca1cdddfc3c042e70bf3117\": container with ID starting with 2413071844b82bf1f0a0c36fc99c76cbfd94778e0ca1cdddfc3c042e70bf3117 not found: ID does not exist"
Nov 03 21:51:19.501136 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.501104    5053 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f68c476-0110-4dd1-a322-e7f0e991f56b-local-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:19.501136 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.501135    5053 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-csr-dns-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:19.501526 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.501148    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnw6g\" (UniqueName: \"kubernetes.io/projected/7f68c476-0110-4dd1-a322-e7f0e991f56b-kube-api-access-dnw6g\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:19.501526 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.501228    5053 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-kubeconfig\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:19.501526 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.501267    5053 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7f68c476-0110-4dd1-a322-e7f0e991f56b-cacerts\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:19.501526 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.501278    5053 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-csr-ca-configmap\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:19.501526 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.501291    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f68c476-0110-4dd1-a322-e7f0e991f56b-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:19.633686 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.633643    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"]
Nov 03 21:51:19.636515 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.636487    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-24g96"]
Nov 03 21:51:19.667742 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.667703    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a570d7ee-ec67-4db1-a2d2-0ffc18b0d439" path="/var/lib/kubelet/pods/a570d7ee-ec67-4db1-a2d2-0ffc18b0d439/volumes"
Nov 03 21:51:19.743204 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.743120    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"]
Nov 03 21:51:19.747218 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:19.747189    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8xj6"]
Nov 03 21:51:21.668371 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:21.668336    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f68c476-0110-4dd1-a322-e7f0e991f56b" path="/var/lib/kubelet/pods/7f68c476-0110-4dd1-a322-e7f0e991f56b/volumes"
Nov 03 21:51:32.638046 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:32.637950    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"]
Nov 03 21:51:32.638562 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:32.638225    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7" podUID="986245b5-a0a6-4997-92aa-3b1da76ac041" containerName="maas-api" containerID="cri-o://16ebd88658d6ddaf541ca288f7d628ea6e671345672c23c1774aea6b62e39628" gracePeriod=30
Nov 03 21:51:32.938055 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:32.938023    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"
Nov 03 21:51:33.076650 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:33.076619    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/986245b5-a0a6-4997-92aa-3b1da76ac041-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/986245b5-a0a6-4997-92aa-3b1da76ac041-openshift-service-ca\") pod \"986245b5-a0a6-4997-92aa-3b1da76ac041\" (UID: \"986245b5-a0a6-4997-92aa-3b1da76ac041\") "
Nov 03 21:51:33.076835 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:33.076724    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/986245b5-a0a6-4997-92aa-3b1da76ac041-kube-api-access-8g2nx\" (UniqueName: \"kubernetes.io/projected/986245b5-a0a6-4997-92aa-3b1da76ac041-kube-api-access-8g2nx\") pod \"986245b5-a0a6-4997-92aa-3b1da76ac041\" (UID: \"986245b5-a0a6-4997-92aa-3b1da76ac041\") "
Nov 03 21:51:33.076903 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:33.076851    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/986245b5-a0a6-4997-92aa-3b1da76ac041-metrics-certs\" (UniqueName: \"kubernetes.io/secret/986245b5-a0a6-4997-92aa-3b1da76ac041-metrics-certs\") pod \"986245b5-a0a6-4997-92aa-3b1da76ac041\" (UID: \"986245b5-a0a6-4997-92aa-3b1da76ac041\") "
Nov 03 21:51:33.076958 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:33.076931    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/986245b5-a0a6-4997-92aa-3b1da76ac041-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/986245b5-a0a6-4997-92aa-3b1da76ac041-maas-api-tls\") pod \"986245b5-a0a6-4997-92aa-3b1da76ac041\" (UID: \"986245b5-a0a6-4997-92aa-3b1da76ac041\") "
Nov 03 21:51:33.077524 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:33.077102    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/986245b5-a0a6-4997-92aa-3b1da76ac041-openshift-service-ca" pod "986245b5-a0a6-4997-92aa-3b1da76ac041" (UID: "986245b5-a0a6-4997-92aa-3b1da76ac041"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:51:33.077524 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:33.077481    5053 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/986245b5-a0a6-4997-92aa-3b1da76ac041-openshift-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:33.080071 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:33.080044    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/986245b5-a0a6-4997-92aa-3b1da76ac041-kube-api-access-8g2nx" pod "986245b5-a0a6-4997-92aa-3b1da76ac041" (UID: "986245b5-a0a6-4997-92aa-3b1da76ac041"). InnerVolumeSpecName "kube-api-access-8g2nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:51:33.080888 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:33.080862    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/986245b5-a0a6-4997-92aa-3b1da76ac041-metrics-certs" pod "986245b5-a0a6-4997-92aa-3b1da76ac041" (UID: "986245b5-a0a6-4997-92aa-3b1da76ac041"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:51:33.082580 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:33.082559    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/986245b5-a0a6-4997-92aa-3b1da76ac041-maas-api-tls" pod "986245b5-a0a6-4997-92aa-3b1da76ac041" (UID: "986245b5-a0a6-4997-92aa-3b1da76ac041"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:51:33.178823 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:33.178738    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/986245b5-a0a6-4997-92aa-3b1da76ac041-maas-api-tls\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:33.178823 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:33.178771    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g2nx\" (UniqueName: \"kubernetes.io/projected/986245b5-a0a6-4997-92aa-3b1da76ac041-kube-api-access-8g2nx\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:33.178823 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:33.178780    5053 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/986245b5-a0a6-4997-92aa-3b1da76ac041-metrics-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:33.538715 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:33.538682    5053 generic.go:356] "Generic (PLEG): container finished" podID="986245b5-a0a6-4997-92aa-3b1da76ac041" containerID="16ebd88658d6ddaf541ca288f7d628ea6e671345672c23c1774aea6b62e39628" exitCode=0
Nov 03 21:51:33.538921 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:33.538731    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"
Nov 03 21:51:33.538921 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:33.538754    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7" event={"ID":"986245b5-a0a6-4997-92aa-3b1da76ac041","Type":"ContainerDied","Data":"16ebd88658d6ddaf541ca288f7d628ea6e671345672c23c1774aea6b62e39628"}
Nov 03 21:51:33.538921 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:33.538807    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7" event={"ID":"986245b5-a0a6-4997-92aa-3b1da76ac041","Type":"ContainerDied","Data":"1260ba2fd17613a475a9e29c7b08dc955aa58a11f6a9752fbf98446d81584b87"}
Nov 03 21:51:33.538921 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:33.538830    5053 scope.go:122] "RemoveContainer" containerID="16ebd88658d6ddaf541ca288f7d628ea6e671345672c23c1774aea6b62e39628"
Nov 03 21:51:33.549924 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:33.549905    5053 scope.go:122] "RemoveContainer" containerID="16ebd88658d6ddaf541ca288f7d628ea6e671345672c23c1774aea6b62e39628"
Nov 03 21:51:33.550353 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:51:33.550229    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16ebd88658d6ddaf541ca288f7d628ea6e671345672c23c1774aea6b62e39628\": container with ID starting with 16ebd88658d6ddaf541ca288f7d628ea6e671345672c23c1774aea6b62e39628 not found: ID does not exist" containerID="16ebd88658d6ddaf541ca288f7d628ea6e671345672c23c1774aea6b62e39628"
Nov 03 21:51:33.550451 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:33.550365    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ebd88658d6ddaf541ca288f7d628ea6e671345672c23c1774aea6b62e39628"} err="failed to get container status \"16ebd88658d6ddaf541ca288f7d628ea6e671345672c23c1774aea6b62e39628\": rpc error: code = NotFound desc = could not find container \"16ebd88658d6ddaf541ca288f7d628ea6e671345672c23c1774aea6b62e39628\": container with ID starting with 16ebd88658d6ddaf541ca288f7d628ea6e671345672c23c1774aea6b62e39628 not found: ID does not exist"
Nov 03 21:51:33.726051 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:33.726015    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"]
Nov 03 21:51:33.758984 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:33.758948    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c0bdb-655bf75c5-9skc7"]
Nov 03 21:51:35.676418 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:35.676387    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="986245b5-a0a6-4997-92aa-3b1da76ac041" path="/var/lib/kubelet/pods/986245b5-a0a6-4997-92aa-3b1da76ac041/volumes"
Nov 03 21:51:40.929096 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:51:40.929058    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:51:41.466556 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:41.466515    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"]
Nov 03 21:51:41.466819 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:41.466774    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2" podUID="1dc20893-5839-4b09-b930-9b9f150105fa" containerName="istio-proxy" containerID="cri-o://2dc2caa1db2715c43516bb5ba922b388a2d9d99471b9a76f2eaf42b2c912b745" gracePeriod=30
Nov 03 21:51:43.001206 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:43.001166    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"]
Nov 03 21:51:43.001730 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:43.001463    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd" podUID="7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" containerName="istio-proxy" containerID="cri-o://0981d3f8d9298c3008c7367fb67fab6719327572a0555af81b81de3e2e30b01e" gracePeriod=30
Nov 03 21:51:43.170307 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:43.170186    5053 patch_prober.go:29] interesting pod/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:51:43.170307 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:43.170281    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2" podUID="1dc20893-5839-4b09-b930-9b9f150105fa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:51:43.611147 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:43.611100    5053 patch_prober.go:29] interesting pod/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:51:43.611356 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:43.611160    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd" podUID="7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:51:45.157362 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.157307    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"]
Nov 03 21:51:45.158067 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.158047    5053 cpu_manager.go:414] "containerMap: removing container" podUID="5d7bc971-f569-4f7c-88bf-5e3168bd49bc" containerName="istio-proxy"
Nov 03 21:51:45.158121 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.158069    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d7bc971-f569-4f7c-88bf-5e3168bd49bc" containerName="istio-proxy"
Nov 03 21:51:45.158121 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.158086    5053 cpu_manager.go:414] "containerMap: removing container" podUID="986245b5-a0a6-4997-92aa-3b1da76ac041" containerName="maas-api"
Nov 03 21:51:45.158121 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.158091    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="986245b5-a0a6-4997-92aa-3b1da76ac041" containerName="maas-api"
Nov 03 21:51:45.158270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.158125    5053 cpu_manager.go:414] "containerMap: removing container" podUID="a570d7ee-ec67-4db1-a2d2-0ffc18b0d439" containerName="istio-proxy"
Nov 03 21:51:45.158270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.158131    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a570d7ee-ec67-4db1-a2d2-0ffc18b0d439" containerName="istio-proxy"
Nov 03 21:51:45.158270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.158141    5053 cpu_manager.go:414] "containerMap: removing container" podUID="7f68c476-0110-4dd1-a322-e7f0e991f56b" containerName="discovery"
Nov 03 21:51:45.158270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.158146    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f68c476-0110-4dd1-a322-e7f0e991f56b" containerName="discovery"
Nov 03 21:51:45.158270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.158249    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d7bc971-f569-4f7c-88bf-5e3168bd49bc" containerName="istio-proxy"
Nov 03 21:51:45.158270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.158262    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="986245b5-a0a6-4997-92aa-3b1da76ac041" containerName="maas-api"
Nov 03 21:51:45.158474 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.158279    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="a570d7ee-ec67-4db1-a2d2-0ffc18b0d439" containerName="istio-proxy"
Nov 03 21:51:45.158474 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.158286    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f68c476-0110-4dd1-a322-e7f0e991f56b" containerName="discovery"
Nov 03 21:51:45.162169 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.162152    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.214609 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.214578    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-ca758d-openshift-default-dockercfg-4mt9b\""
Nov 03 21:51:45.287559 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.287500    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"]
Nov 03 21:51:45.369448 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.369407    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-podinfo\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.369448 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.369447    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-token\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.369701 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.369485    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-credential-socket\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.369701 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.369582    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istiod-ca-cert\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.369701 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.369641    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-workload-socket\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.369701 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.369673    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-data\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.369910 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.369782    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-workload-certs\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.369910 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.369840    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swxcc\" (UniqueName: \"kubernetes.io/projected/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-kube-api-access-swxcc\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.370015 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.369932    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-envoy\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.471174 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.471139    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swxcc\" (UniqueName: \"kubernetes.io/projected/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-kube-api-access-swxcc\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.471414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.471221    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-envoy\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.471414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.471317    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-podinfo\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.471414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.471335    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-token\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.471414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.471354    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-credential-socket\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.471414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.471374    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istiod-ca-cert\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.471710 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.471433    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-workload-socket\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.471710 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.471469    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-data\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.471710 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.471527    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-workload-certs\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.471910 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.471889    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-workload-certs\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.472268 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.472218    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-workload-socket\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.472365 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.472279    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-data\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.472365 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.472284    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istiod-ca-cert\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.472437 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.472415    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-credential-socket\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.474150 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.474124    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-envoy\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.474353 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.474336    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-podinfo\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.528465 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.528431    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-token\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.528659 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.528635    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swxcc\" (UniqueName: \"kubernetes.io/projected/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-kube-api-access-swxcc\") pod \"e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") " pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:45.832706 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.832602    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-ca758d-openshift-default-dockercfg-4mt9b\""
Nov 03 21:51:45.842137 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:45.842111    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:46.282758 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.282722    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"]
Nov 03 21:51:46.320960 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:51:46.320929    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0510a33_96d0_4ad8_a5f3_3dba5a009fc7.slice/crio-d3df071e6877371851e4e24d912e594c8a0959d4c6255e4d7f21f0483b1df7d3 WatchSource:0}: Error finding container d3df071e6877371851e4e24d912e594c8a0959d4c6255e4d7f21f0483b1df7d3: Status 404 returned error can't find the container with id d3df071e6877371851e4e24d912e594c8a0959d4c6255e4d7f21f0483b1df7d3
Nov 03 21:51:46.323478 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.323443    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:51:46.323563 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.323513    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:51:46.323563 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.323541    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:51:46.670818 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.670768    5053 generic.go:356] "Generic (PLEG): container finished" podID="1dc20893-5839-4b09-b930-9b9f150105fa" containerID="2dc2caa1db2715c43516bb5ba922b388a2d9d99471b9a76f2eaf42b2c912b745" exitCode=0
Nov 03 21:51:46.671030 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.670838    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2" event={"ID":"1dc20893-5839-4b09-b930-9b9f150105fa","Type":"ContainerDied","Data":"2dc2caa1db2715c43516bb5ba922b388a2d9d99471b9a76f2eaf42b2c912b745"}
Nov 03 21:51:46.671030 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.670905    5053 scope.go:122] "RemoveContainer" containerID="35b65990e60f7449e382b9e2e3b20ffef53706f86b55874c33f94d6cd0dd9a51"
Nov 03 21:51:46.677851 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.677815    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4" event={"ID":"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7","Type":"ContainerStarted","Data":"9c350f8575b4e372536a24c13a1a58240f6d99afbfc8149d4f4ca0cabcff5078"}
Nov 03 21:51:46.678133 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.678095    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4" event={"ID":"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7","Type":"ContainerStarted","Data":"d3df071e6877371851e4e24d912e594c8a0959d4c6255e4d7f21f0483b1df7d3"}
Nov 03 21:51:46.762984 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.762958    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:51:46.797627 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.797564    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4" podStartSLOduration=1.797546805 podStartE2EDuration="1.797546805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:51:46.795666346 +0000 UTC m=+2221.289588219" watchObservedRunningTime="2026-11-03 21:51:46.797546805 +0000 UTC m=+2221.291468681"
Nov 03 21:51:46.843125 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.843055    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:46.898995 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.898961    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1dc20893-5839-4b09-b930-9b9f150105fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dc20893-5839-4b09-b930-9b9f150105fa-istio-podinfo\") pod \"1dc20893-5839-4b09-b930-9b9f150105fa\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") "
Nov 03 21:51:46.899161 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.899034    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-credential-socket\") pod \"1dc20893-5839-4b09-b930-9b9f150105fa\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") "
Nov 03 21:51:46.899161 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.899131    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-workload-socket\") pod \"1dc20893-5839-4b09-b930-9b9f150105fa\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") "
Nov 03 21:51:46.899303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.899209    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-workload-certs\") pod \"1dc20893-5839-4b09-b930-9b9f150105fa\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") "
Nov 03 21:51:46.899303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.899276    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-istio-envoy\") pod \"1dc20893-5839-4b09-b930-9b9f150105fa\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") "
Nov 03 21:51:46.899398 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.899361    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dc20893-5839-4b09-b930-9b9f150105fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dc20893-5839-4b09-b930-9b9f150105fa-istiod-ca-cert\") pod \"1dc20893-5839-4b09-b930-9b9f150105fa\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") "
Nov 03 21:51:46.899398 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.899368    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-credential-socket" pod "1dc20893-5839-4b09-b930-9b9f150105fa" (UID: "1dc20893-5839-4b09-b930-9b9f150105fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:46.899511 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.899391    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-workload-socket" pod "1dc20893-5839-4b09-b930-9b9f150105fa" (UID: "1dc20893-5839-4b09-b930-9b9f150105fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:46.899511 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.899403    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dc20893-5839-4b09-b930-9b9f150105fa-kube-api-access-64lcw\" (UniqueName: \"kubernetes.io/projected/1dc20893-5839-4b09-b930-9b9f150105fa-kube-api-access-64lcw\") pod \"1dc20893-5839-4b09-b930-9b9f150105fa\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") "
Nov 03 21:51:46.899511 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.899453    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-istio-data\") pod \"1dc20893-5839-4b09-b930-9b9f150105fa\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") "
Nov 03 21:51:46.899511 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.899454    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-workload-certs" pod "1dc20893-5839-4b09-b930-9b9f150105fa" (UID: "1dc20893-5839-4b09-b930-9b9f150105fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:46.899511 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.899506    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dc20893-5839-4b09-b930-9b9f150105fa-istio-token\" (UniqueName: \"kubernetes.io/projected/1dc20893-5839-4b09-b930-9b9f150105fa-istio-token\") pod \"1dc20893-5839-4b09-b930-9b9f150105fa\" (UID: \"1dc20893-5839-4b09-b930-9b9f150105fa\") "
Nov 03 21:51:46.899982 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.899679    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-istio-data" pod "1dc20893-5839-4b09-b930-9b9f150105fa" (UID: "1dc20893-5839-4b09-b930-9b9f150105fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:46.899982 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.899716    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc20893-5839-4b09-b930-9b9f150105fa-istiod-ca-cert" pod "1dc20893-5839-4b09-b930-9b9f150105fa" (UID: "1dc20893-5839-4b09-b930-9b9f150105fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:51:46.900112 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.900051    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:46.900112 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.900072    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:46.900112 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.900088    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:46.900112 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.900101    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dc20893-5839-4b09-b930-9b9f150105fa-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:46.900282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.900114    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:46.901554 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.901526    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1dc20893-5839-4b09-b930-9b9f150105fa-istio-podinfo" pod "1dc20893-5839-4b09-b930-9b9f150105fa" (UID: "1dc20893-5839-4b09-b930-9b9f150105fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:51:46.901554 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.901542    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-istio-envoy" pod "1dc20893-5839-4b09-b930-9b9f150105fa" (UID: "1dc20893-5839-4b09-b930-9b9f150105fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:46.902141 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.902122    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc20893-5839-4b09-b930-9b9f150105fa-kube-api-access-64lcw" pod "1dc20893-5839-4b09-b930-9b9f150105fa" (UID: "1dc20893-5839-4b09-b930-9b9f150105fa"). InnerVolumeSpecName "kube-api-access-64lcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:51:46.902219 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:46.902200    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc20893-5839-4b09-b930-9b9f150105fa-istio-token" pod "1dc20893-5839-4b09-b930-9b9f150105fa" (UID: "1dc20893-5839-4b09-b930-9b9f150105fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:51:47.001730 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:47.001694    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dc20893-5839-4b09-b930-9b9f150105fa-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:47.001730 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:47.001728    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64lcw\" (UniqueName: \"kubernetes.io/projected/1dc20893-5839-4b09-b930-9b9f150105fa-kube-api-access-64lcw\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:47.001937 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:47.001741    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dc20893-5839-4b09-b930-9b9f150105fa-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:47.001937 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:47.001753    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dc20893-5839-4b09-b930-9b9f150105fa-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:47.685299 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:47.685267    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2" event={"ID":"1dc20893-5839-4b09-b930-9b9f150105fa","Type":"ContainerDied","Data":"ed7f8fd35ea18b915c76840b40c246b9fc0b6bf8cb6ddd565debc786c09c878c"}
Nov 03 21:51:47.685696 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:47.685337    5053 scope.go:122] "RemoveContainer" containerID="2dc2caa1db2715c43516bb5ba922b388a2d9d99471b9a76f2eaf42b2c912b745"
Nov 03 21:51:47.685696 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:47.685276    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"
Nov 03 21:51:47.858342 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:47.849695    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:47.898617 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:47.898576    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"]
Nov 03 21:51:47.950271 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:47.950199    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c0bdb-openshift-default-7d944d7687-smvf2"]
Nov 03 21:51:48.304126 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.304100    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:51:48.326093 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.326062    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-data\") pod \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") "
Nov 03 21:51:48.326253 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.326106    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-workload-socket\") pod \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") "
Nov 03 21:51:48.326253 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.326137    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-envoy\") pod \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") "
Nov 03 21:51:48.326253 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.326207    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-token\" (UniqueName: \"kubernetes.io/projected/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-token\") pod \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") "
Nov 03 21:51:48.326443 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.326334    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-data" pod "7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" (UID: "7f60ff6a-7db5-43cb-ac4c-c9d287139cf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:48.326443 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.326372    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-kube-api-access-scvkq\" (UniqueName: \"kubernetes.io/projected/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-kube-api-access-scvkq\") pod \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") "
Nov 03 21:51:48.326556 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.326449    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-workload-socket" pod "7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" (UID: "7f60ff6a-7db5-43cb-ac4c-c9d287139cf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:48.326556 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.326499    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istiod-ca-cert\") pod \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") "
Nov 03 21:51:48.326661 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.326555    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-workload-certs\") pod \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") "
Nov 03 21:51:48.326661 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.326631    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-credential-socket\") pod \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") "
Nov 03 21:51:48.326761 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.326731    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-podinfo\") pod \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\" (UID: \"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3\") "
Nov 03 21:51:48.327167 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.326877    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istiod-ca-cert" pod "7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" (UID: "7f60ff6a-7db5-43cb-ac4c-c9d287139cf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:51:48.327167 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.326985    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-credential-socket" pod "7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" (UID: "7f60ff6a-7db5-43cb-ac4c-c9d287139cf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:48.327587 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.327271    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:48.327587 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.327292    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:48.327587 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.327306    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:48.327587 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.327319    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:48.327587 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.327537    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-workload-certs" pod "7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" (UID: "7f60ff6a-7db5-43cb-ac4c-c9d287139cf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:48.328818 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.328792    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-kube-api-access-scvkq" pod "7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" (UID: "7f60ff6a-7db5-43cb-ac4c-c9d287139cf3"). InnerVolumeSpecName "kube-api-access-scvkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:51:48.328955 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.328934    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-envoy" pod "7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" (UID: "7f60ff6a-7db5-43cb-ac4c-c9d287139cf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:48.329462 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.329437    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-token" pod "7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" (UID: "7f60ff6a-7db5-43cb-ac4c-c9d287139cf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:51:48.329804 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.329786    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-podinfo" pod "7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" (UID: "7f60ff6a-7db5-43cb-ac4c-c9d287139cf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:51:48.428427 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.428390    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:48.428427 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.428427    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:48.428615 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.428444    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scvkq\" (UniqueName: \"kubernetes.io/projected/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-kube-api-access-scvkq\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:48.428615 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.428459    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:48.428615 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.428472    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:48.693557 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.693523    5053 generic.go:356] "Generic (PLEG): container finished" podID="7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" containerID="0981d3f8d9298c3008c7367fb67fab6719327572a0555af81b81de3e2e30b01e" exitCode=0
Nov 03 21:51:48.694010 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.693569    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"
Nov 03 21:51:48.694010 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.693606    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd" event={"ID":"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3","Type":"ContainerDied","Data":"0981d3f8d9298c3008c7367fb67fab6719327572a0555af81b81de3e2e30b01e"}
Nov 03 21:51:48.694010 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.693653    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd" event={"ID":"7f60ff6a-7db5-43cb-ac4c-c9d287139cf3","Type":"ContainerDied","Data":"8eecca20b40767a06e2a91fc70d49ad2d3b2ae838b229bb80a6a4f23af1e671e"}
Nov 03 21:51:48.694010 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.693679    5053 scope.go:122] "RemoveContainer" containerID="0981d3f8d9298c3008c7367fb67fab6719327572a0555af81b81de3e2e30b01e"
Nov 03 21:51:48.695187 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.695168    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:48.696377 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.696357    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:51:48.729992 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.729954    5053 scope.go:122] "RemoveContainer" containerID="80445fe3e688fc4de4fb5df38d984dc8c251ee1407fedc5c9e9bf675362a5a34"
Nov 03 21:51:48.744461 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.744421    5053 scope.go:122] "RemoveContainer" containerID="0981d3f8d9298c3008c7367fb67fab6719327572a0555af81b81de3e2e30b01e"
Nov 03 21:51:48.744898 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:51:48.744871    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0981d3f8d9298c3008c7367fb67fab6719327572a0555af81b81de3e2e30b01e\": container with ID starting with 0981d3f8d9298c3008c7367fb67fab6719327572a0555af81b81de3e2e30b01e not found: ID does not exist" containerID="0981d3f8d9298c3008c7367fb67fab6719327572a0555af81b81de3e2e30b01e"
Nov 03 21:51:48.745015 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.744911    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0981d3f8d9298c3008c7367fb67fab6719327572a0555af81b81de3e2e30b01e"} err="failed to get container status \"0981d3f8d9298c3008c7367fb67fab6719327572a0555af81b81de3e2e30b01e\": rpc error: code = NotFound desc = could not find container \"0981d3f8d9298c3008c7367fb67fab6719327572a0555af81b81de3e2e30b01e\": container with ID starting with 0981d3f8d9298c3008c7367fb67fab6719327572a0555af81b81de3e2e30b01e not found: ID does not exist"
Nov 03 21:51:48.745015 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.744936    5053 scope.go:122] "RemoveContainer" containerID="80445fe3e688fc4de4fb5df38d984dc8c251ee1407fedc5c9e9bf675362a5a34"
Nov 03 21:51:48.745376 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:51:48.745350    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80445fe3e688fc4de4fb5df38d984dc8c251ee1407fedc5c9e9bf675362a5a34\": container with ID starting with 80445fe3e688fc4de4fb5df38d984dc8c251ee1407fedc5c9e9bf675362a5a34 not found: ID does not exist" containerID="80445fe3e688fc4de4fb5df38d984dc8c251ee1407fedc5c9e9bf675362a5a34"
Nov 03 21:51:48.745602 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.745386    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80445fe3e688fc4de4fb5df38d984dc8c251ee1407fedc5c9e9bf675362a5a34"} err="failed to get container status \"80445fe3e688fc4de4fb5df38d984dc8c251ee1407fedc5c9e9bf675362a5a34\": rpc error: code = NotFound desc = could not find container \"80445fe3e688fc4de4fb5df38d984dc8c251ee1407fedc5c9e9bf675362a5a34\": container with ID starting with 80445fe3e688fc4de4fb5df38d984dc8c251ee1407fedc5c9e9bf675362a5a34 not found: ID does not exist"
Nov 03 21:51:48.858585 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.858541    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"]
Nov 03 21:51:48.861288 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:48.861269    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-3c6597-openshift-default-6b6fd5c969-xbkdd"]
Nov 03 21:51:49.668831 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:49.668802    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dc20893-5839-4b09-b930-9b9f150105fa" path="/var/lib/kubelet/pods/1dc20893-5839-4b09-b930-9b9f150105fa/volumes"
Nov 03 21:51:49.669309 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:49.669296    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" path="/var/lib/kubelet/pods/7f60ff6a-7db5-43cb-ac4c-c9d287139cf3/volumes"
Nov 03 21:51:55.467193 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.467148    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx"]
Nov 03 21:51:55.467762 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.467746    5053 cpu_manager.go:414] "containerMap: removing container" podUID="1dc20893-5839-4b09-b930-9b9f150105fa" containerName="istio-proxy"
Nov 03 21:51:55.467762 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.467761    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc20893-5839-4b09-b930-9b9f150105fa" containerName="istio-proxy"
Nov 03 21:51:55.467892 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.467778    5053 cpu_manager.go:414] "containerMap: removing container" podUID="7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" containerName="istio-proxy"
Nov 03 21:51:55.467892 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.467785    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" containerName="istio-proxy"
Nov 03 21:51:55.467892 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.467817    5053 cpu_manager.go:414] "containerMap: removing container" podUID="1dc20893-5839-4b09-b930-9b9f150105fa" containerName="istio-proxy"
Nov 03 21:51:55.467892 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.467823    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc20893-5839-4b09-b930-9b9f150105fa" containerName="istio-proxy"
Nov 03 21:51:55.467892 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.467837    5053 cpu_manager.go:414] "containerMap: removing container" podUID="7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" containerName="istio-proxy"
Nov 03 21:51:55.467892 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.467842    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" containerName="istio-proxy"
Nov 03 21:51:55.468195 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.467948    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc20893-5839-4b09-b930-9b9f150105fa" containerName="istio-proxy"
Nov 03 21:51:55.468195 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.467956    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" containerName="istio-proxy"
Nov 03 21:51:55.468195 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.467965    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f60ff6a-7db5-43cb-ac4c-c9d287139cf3" containerName="istio-proxy"
Nov 03 21:51:55.473091 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.473065    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx"
Nov 03 21:51:55.525296 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.525265    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dgkr\" (UniqueName: \"kubernetes.io/projected/5c79e9ce-b577-4034-8943-78ef2b0134e2-kube-api-access-2dgkr\") pod \"payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx\" (UID: \"5c79e9ce-b577-4034-8943-78ef2b0134e2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx"
Nov 03 21:51:55.525677 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.525652    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c79e9ce-b577-4034-8943-78ef2b0134e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx\" (UID: \"5c79e9ce-b577-4034-8943-78ef2b0134e2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx"
Nov 03 21:51:55.527883 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.527852    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx"]
Nov 03 21:51:55.583160 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.581665    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-ca758d-dockercfg-pq2gg\""
Nov 03 21:51:55.583160 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.581801    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-ca758d\""
Nov 03 21:51:55.636525 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.636485    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dgkr\" (UniqueName: \"kubernetes.io/projected/5c79e9ce-b577-4034-8943-78ef2b0134e2-kube-api-access-2dgkr\") pod \"payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx\" (UID: \"5c79e9ce-b577-4034-8943-78ef2b0134e2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx"
Nov 03 21:51:55.636727 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.636604    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c79e9ce-b577-4034-8943-78ef2b0134e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx\" (UID: \"5c79e9ce-b577-4034-8943-78ef2b0134e2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx"
Nov 03 21:51:55.668972 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.668937    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c79e9ce-b577-4034-8943-78ef2b0134e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx\" (UID: \"5c79e9ce-b577-4034-8943-78ef2b0134e2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx"
Nov 03 21:51:55.679719 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.679689    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn"]
Nov 03 21:51:55.680373 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.680359    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc20893-5839-4b09-b930-9b9f150105fa" containerName="istio-proxy"
Nov 03 21:51:55.683938 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.683858    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn"
Nov 03 21:51:55.688026 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.688004    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn"]
Nov 03 21:51:55.700932 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.700905    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dgkr\" (UniqueName: \"kubernetes.io/projected/5c79e9ce-b577-4034-8943-78ef2b0134e2-kube-api-access-2dgkr\") pod \"payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx\" (UID: \"5c79e9ce-b577-4034-8943-78ef2b0134e2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx"
Nov 03 21:51:55.737456 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.737377    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk47m\" (UniqueName: \"kubernetes.io/projected/4ff958dd-42eb-4f29-88c4-f22e849bac96-kube-api-access-fk47m\") pod \"payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn\" (UID: \"4ff958dd-42eb-4f29-88c4-f22e849bac96\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn"
Nov 03 21:51:55.737456 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.737444    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ff958dd-42eb-4f29-88c4-f22e849bac96-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn\" (UID: \"4ff958dd-42eb-4f29-88c4-f22e849bac96\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn"
Nov 03 21:51:55.786999 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.786966    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx"
Nov 03 21:51:55.838423 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.838390    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ff958dd-42eb-4f29-88c4-f22e849bac96-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn\" (UID: \"4ff958dd-42eb-4f29-88c4-f22e849bac96\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn"
Nov 03 21:51:55.838601 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.838544    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk47m\" (UniqueName: \"kubernetes.io/projected/4ff958dd-42eb-4f29-88c4-f22e849bac96-kube-api-access-fk47m\") pod \"payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn\" (UID: \"4ff958dd-42eb-4f29-88c4-f22e849bac96\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn"
Nov 03 21:51:55.839071 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.839043    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ff958dd-42eb-4f29-88c4-f22e849bac96-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn\" (UID: \"4ff958dd-42eb-4f29-88c4-f22e849bac96\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn"
Nov 03 21:51:55.896139 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.896107    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk47m\" (UniqueName: \"kubernetes.io/projected/4ff958dd-42eb-4f29-88c4-f22e849bac96-kube-api-access-fk47m\") pod \"payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn\" (UID: \"4ff958dd-42eb-4f29-88c4-f22e849bac96\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn"
Nov 03 21:51:55.999546 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:55.999451    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn"
Nov 03 21:51:56.166605 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:56.166580    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx"]
Nov 03 21:51:56.167630 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:51:56.167607    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c79e9ce_b577_4034_8943_78ef2b0134e2.slice/crio-28a80c71ac012bd8840e0e974ef381deb3e19eaa3d7be786837b6193b4272c5e WatchSource:0}: Error finding container 28a80c71ac012bd8840e0e974ef381deb3e19eaa3d7be786837b6193b4272c5e: Status 404 returned error can't find the container with id 28a80c71ac012bd8840e0e974ef381deb3e19eaa3d7be786837b6193b4272c5e
Nov 03 21:51:56.379633 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:56.379584    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn"]
Nov 03 21:51:56.380733 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:51:56.380704    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff958dd_42eb_4f29_88c4_f22e849bac96.slice/crio-5170d67ec691d240137fc451d701de3e7cf2e37729dfbe9e68784ae0524f4660 WatchSource:0}: Error finding container 5170d67ec691d240137fc451d701de3e7cf2e37729dfbe9e68784ae0524f4660: Status 404 returned error can't find the container with id 5170d67ec691d240137fc451d701de3e7cf2e37729dfbe9e68784ae0524f4660
Nov 03 21:51:56.766787 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:56.766749    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn" event={"ID":"4ff958dd-42eb-4f29-88c4-f22e849bac96","Type":"ContainerStarted","Data":"e8b4d1d871bc96984858d94142dc501afa43aa181bcc5c6c1c185af88211036f"}
Nov 03 21:51:56.767273 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:56.766837    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn" event={"ID":"4ff958dd-42eb-4f29-88c4-f22e849bac96","Type":"ContainerStarted","Data":"5170d67ec691d240137fc451d701de3e7cf2e37729dfbe9e68784ae0524f4660"}
Nov 03 21:51:56.767273 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:56.766887    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn"
Nov 03 21:51:56.768414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:56.768388    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx" event={"ID":"5c79e9ce-b577-4034-8943-78ef2b0134e2","Type":"ContainerStarted","Data":"992eb0c4010421d86a8fa0630c0558c6d7e42a35418fc2e54390c81c7ae185be"}
Nov 03 21:51:56.768543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:56.768437    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx" event={"ID":"5c79e9ce-b577-4034-8943-78ef2b0134e2","Type":"ContainerStarted","Data":"28a80c71ac012bd8840e0e974ef381deb3e19eaa3d7be786837b6193b4272c5e"}
Nov 03 21:51:56.768543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:56.768467    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx"
Nov 03 21:51:56.880945 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:56.880879    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn" podStartSLOduration=1.880864708 podStartE2EDuration="1.880864708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:51:56.880465965 +0000 UTC m=+2231.374387840" watchObservedRunningTime="2026-11-03 21:51:56.880864708 +0000 UTC m=+2231.374786584"
Nov 03 21:51:57.007751 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:51:57.002068    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx" podStartSLOduration=2.002044687 podStartE2EDuration="2.002044687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:51:56.995369387 +0000 UTC m=+2231.489291276" watchObservedRunningTime="2026-11-03 21:51:57.002044687 +0000 UTC m=+2231.495966562"
Nov 03 21:52:07.778863 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:07.778825    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn"
Nov 03 21:52:07.779318 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:07.778900    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx"
Nov 03 21:52:10.927804 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:52:10.927757    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:52:40.929927 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:52:40.929884    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:52:51.682099 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:51.682064    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"]
Nov 03 21:52:51.682550 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:51.682283    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9" podUID="1e6dbdf0-e267-42b5-a916-4f39502de63d" containerName="maas-api" containerID="cri-o://b82a4debb44d1ef7340e5b4fafd93a7999472fb0bf1c36f04a2b0110d8aca917" gracePeriod=30
Nov 03 21:52:51.992272 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:51.992212    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"
Nov 03 21:52:52.045635 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.045594    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e6dbdf0-e267-42b5-a916-4f39502de63d-kube-api-access-tnrmt\" (UniqueName: \"kubernetes.io/projected/1e6dbdf0-e267-42b5-a916-4f39502de63d-kube-api-access-tnrmt\") pod \"1e6dbdf0-e267-42b5-a916-4f39502de63d\" (UID: \"1e6dbdf0-e267-42b5-a916-4f39502de63d\") "
Nov 03 21:52:52.045859 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.045797    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e6dbdf0-e267-42b5-a916-4f39502de63d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e6dbdf0-e267-42b5-a916-4f39502de63d-maas-api-tls\") pod \"1e6dbdf0-e267-42b5-a916-4f39502de63d\" (UID: \"1e6dbdf0-e267-42b5-a916-4f39502de63d\") "
Nov 03 21:52:52.045859 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.045838    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e6dbdf0-e267-42b5-a916-4f39502de63d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e6dbdf0-e267-42b5-a916-4f39502de63d-metrics-certs\") pod \"1e6dbdf0-e267-42b5-a916-4f39502de63d\" (UID: \"1e6dbdf0-e267-42b5-a916-4f39502de63d\") "
Nov 03 21:52:52.046033 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.045985    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e6dbdf0-e267-42b5-a916-4f39502de63d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e6dbdf0-e267-42b5-a916-4f39502de63d-openshift-service-ca\") pod \"1e6dbdf0-e267-42b5-a916-4f39502de63d\" (UID: \"1e6dbdf0-e267-42b5-a916-4f39502de63d\") "
Nov 03 21:52:52.046489 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.046425    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e6dbdf0-e267-42b5-a916-4f39502de63d-openshift-service-ca" pod "1e6dbdf0-e267-42b5-a916-4f39502de63d" (UID: "1e6dbdf0-e267-42b5-a916-4f39502de63d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:52:52.047328 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.047291    5053 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e6dbdf0-e267-42b5-a916-4f39502de63d-openshift-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:52:52.048454 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.048432    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e6dbdf0-e267-42b5-a916-4f39502de63d-kube-api-access-tnrmt" pod "1e6dbdf0-e267-42b5-a916-4f39502de63d" (UID: "1e6dbdf0-e267-42b5-a916-4f39502de63d"). InnerVolumeSpecName "kube-api-access-tnrmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:52:52.048454 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.048445    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e6dbdf0-e267-42b5-a916-4f39502de63d-maas-api-tls" pod "1e6dbdf0-e267-42b5-a916-4f39502de63d" (UID: "1e6dbdf0-e267-42b5-a916-4f39502de63d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:52:52.048568 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.048490    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e6dbdf0-e267-42b5-a916-4f39502de63d-metrics-certs" pod "1e6dbdf0-e267-42b5-a916-4f39502de63d" (UID: "1e6dbdf0-e267-42b5-a916-4f39502de63d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:52:52.149007 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.148962    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e6dbdf0-e267-42b5-a916-4f39502de63d-maas-api-tls\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:52:52.149007 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.149007    5053 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e6dbdf0-e267-42b5-a916-4f39502de63d-metrics-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:52:52.149289 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.149022    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnrmt\" (UniqueName: \"kubernetes.io/projected/1e6dbdf0-e267-42b5-a916-4f39502de63d-kube-api-access-tnrmt\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:52:52.168103 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.168053    5053 generic.go:356] "Generic (PLEG): container finished" podID="1e6dbdf0-e267-42b5-a916-4f39502de63d" containerID="b82a4debb44d1ef7340e5b4fafd93a7999472fb0bf1c36f04a2b0110d8aca917" exitCode=0
Nov 03 21:52:52.168103 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.168103    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"
Nov 03 21:52:52.168399 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.168123    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9" event={"ID":"1e6dbdf0-e267-42b5-a916-4f39502de63d","Type":"ContainerDied","Data":"b82a4debb44d1ef7340e5b4fafd93a7999472fb0bf1c36f04a2b0110d8aca917"}
Nov 03 21:52:52.168399 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.168192    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9" event={"ID":"1e6dbdf0-e267-42b5-a916-4f39502de63d","Type":"ContainerDied","Data":"25b9d4233e43050cd482c320c18cd864789b5739056730a9aecedf1601208343"}
Nov 03 21:52:52.168399 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.168222    5053 scope.go:122] "RemoveContainer" containerID="b82a4debb44d1ef7340e5b4fafd93a7999472fb0bf1c36f04a2b0110d8aca917"
Nov 03 21:52:52.179320 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.179222    5053 scope.go:122] "RemoveContainer" containerID="b82a4debb44d1ef7340e5b4fafd93a7999472fb0bf1c36f04a2b0110d8aca917"
Nov 03 21:52:52.179606 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:52:52.179586    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b82a4debb44d1ef7340e5b4fafd93a7999472fb0bf1c36f04a2b0110d8aca917\": container with ID starting with b82a4debb44d1ef7340e5b4fafd93a7999472fb0bf1c36f04a2b0110d8aca917 not found: ID does not exist" containerID="b82a4debb44d1ef7340e5b4fafd93a7999472fb0bf1c36f04a2b0110d8aca917"
Nov 03 21:52:52.179664 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.179612    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b82a4debb44d1ef7340e5b4fafd93a7999472fb0bf1c36f04a2b0110d8aca917"} err="failed to get container status \"b82a4debb44d1ef7340e5b4fafd93a7999472fb0bf1c36f04a2b0110d8aca917\": rpc error: code = NotFound desc = could not find container \"b82a4debb44d1ef7340e5b4fafd93a7999472fb0bf1c36f04a2b0110d8aca917\": container with ID starting with b82a4debb44d1ef7340e5b4fafd93a7999472fb0bf1c36f04a2b0110d8aca917 not found: ID does not exist"
Nov 03 21:52:52.353329 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.349844    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"]
Nov 03 21:52:52.353329 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.351952    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-62cf5c-6cdc6c6fff-4k7t9"]
Nov 03 21:52:52.554705 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.554668    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw"]
Nov 03 21:52:52.555601 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.555582    5053 cpu_manager.go:414] "containerMap: removing container" podUID="1e6dbdf0-e267-42b5-a916-4f39502de63d" containerName="maas-api"
Nov 03 21:52:52.555660 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.555605    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e6dbdf0-e267-42b5-a916-4f39502de63d" containerName="maas-api"
Nov 03 21:52:52.555807 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.555796    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e6dbdf0-e267-42b5-a916-4f39502de63d" containerName="maas-api"
Nov 03 21:52:52.561029 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.561011    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw"
Nov 03 21:52:52.614484 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.612869    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:52:52.614484 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.612934    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:52:52.636945 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.636908    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw"]
Nov 03 21:52:52.654563 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.654526    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfb3e395-5cd3-47bb-b0d2-16cd8958bcae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw\" (UID: \"dfb3e395-5cd3-47bb-b0d2-16cd8958bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw"
Nov 03 21:52:52.654775 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.654602    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfb3e395-5cd3-47bb-b0d2-16cd8958bcae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw\" (UID: \"dfb3e395-5cd3-47bb-b0d2-16cd8958bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw"
Nov 03 21:52:52.654775 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.654694    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjt5x\" (UniqueName: \"kubernetes.io/projected/dfb3e395-5cd3-47bb-b0d2-16cd8958bcae-kube-api-access-gjt5x\") pod \"maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw\" (UID: \"dfb3e395-5cd3-47bb-b0d2-16cd8958bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw"
Nov 03 21:52:52.756024 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.755990    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfb3e395-5cd3-47bb-b0d2-16cd8958bcae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw\" (UID: \"dfb3e395-5cd3-47bb-b0d2-16cd8958bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw"
Nov 03 21:52:52.756548 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.756112    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjt5x\" (UniqueName: \"kubernetes.io/projected/dfb3e395-5cd3-47bb-b0d2-16cd8958bcae-kube-api-access-gjt5x\") pod \"maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw\" (UID: \"dfb3e395-5cd3-47bb-b0d2-16cd8958bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw"
Nov 03 21:52:52.756548 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.756385    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfb3e395-5cd3-47bb-b0d2-16cd8958bcae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw\" (UID: \"dfb3e395-5cd3-47bb-b0d2-16cd8958bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw"
Nov 03 21:52:52.757916 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.757891    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfb3e395-5cd3-47bb-b0d2-16cd8958bcae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw\" (UID: \"dfb3e395-5cd3-47bb-b0d2-16cd8958bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw"
Nov 03 21:52:52.771304 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.759608    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfb3e395-5cd3-47bb-b0d2-16cd8958bcae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw\" (UID: \"dfb3e395-5cd3-47bb-b0d2-16cd8958bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw"
Nov 03 21:52:52.863278 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.858870    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjt5x\" (UniqueName: \"kubernetes.io/projected/dfb3e395-5cd3-47bb-b0d2-16cd8958bcae-kube-api-access-gjt5x\") pod \"maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw\" (UID: \"dfb3e395-5cd3-47bb-b0d2-16cd8958bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw"
Nov 03 21:52:52.876215 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:52.876102    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw"
Nov 03 21:52:53.262998 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:53.262968    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw"]
Nov 03 21:52:53.264341 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:52:53.264313    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb3e395_5cd3_47bb_b0d2_16cd8958bcae.slice/crio-10bb4249bbc345a2ce47a1499834829663a69831d54a3d64d17c566b5e398999 WatchSource:0}: Error finding container 10bb4249bbc345a2ce47a1499834829663a69831d54a3d64d17c566b5e398999: Status 404 returned error can't find the container with id 10bb4249bbc345a2ce47a1499834829663a69831d54a3d64d17c566b5e398999
Nov 03 21:52:53.425435 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:53.425397    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"]
Nov 03 21:52:53.430833 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:53.430814    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"
Nov 03 21:52:53.466783 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:53.466755    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/287dcaa3-959c-4d4c-9fca-490b715ec38d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk\" (UID: \"287dcaa3-959c-4d4c-9fca-490b715ec38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"
Nov 03 21:52:53.466921 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:53.466827    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/287dcaa3-959c-4d4c-9fca-490b715ec38d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk\" (UID: \"287dcaa3-959c-4d4c-9fca-490b715ec38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"
Nov 03 21:52:53.466921 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:53.466888    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287dcaa3-959c-4d4c-9fca-490b715ec38d-metrics-certs\") pod \"maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk\" (UID: \"287dcaa3-959c-4d4c-9fca-490b715ec38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"
Nov 03 21:52:53.467048 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:53.467024    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45hq6\" (UniqueName: \"kubernetes.io/projected/287dcaa3-959c-4d4c-9fca-490b715ec38d-kube-api-access-45hq6\") pod \"maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk\" (UID: \"287dcaa3-959c-4d4c-9fca-490b715ec38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"
Nov 03 21:52:53.483288 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:53.483220    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-ca758d-serving-cert\""
Nov 03 21:52:53.486718 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:53.486692    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"]
Nov 03 21:52:53.568747 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:53.568707    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/287dcaa3-959c-4d4c-9fca-490b715ec38d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk\" (UID: \"287dcaa3-959c-4d4c-9fca-490b715ec38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"
Nov 03 21:52:53.568971 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:53.568828    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/287dcaa3-959c-4d4c-9fca-490b715ec38d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk\" (UID: \"287dcaa3-959c-4d4c-9fca-490b715ec38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"
Nov 03 21:52:53.568971 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:53.568852    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287dcaa3-959c-4d4c-9fca-490b715ec38d-metrics-certs\") pod \"maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk\" (UID: \"287dcaa3-959c-4d4c-9fca-490b715ec38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"
Nov 03 21:52:53.568971 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:52:53.568853    5053 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-serving-cert: secret "maas-api-e2e-praxis-a-ca758d-serving-cert" not found
Nov 03 21:52:53.569140 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:53.569019    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45hq6\" (UniqueName: \"kubernetes.io/projected/287dcaa3-959c-4d4c-9fca-490b715ec38d-kube-api-access-45hq6\") pod \"maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk\" (UID: \"287dcaa3-959c-4d4c-9fca-490b715ec38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"
Nov 03 21:52:53.569140 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:52:53.569086    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/287dcaa3-959c-4d4c-9fca-490b715ec38d-maas-api-tls podName:287dcaa3-959c-4d4c-9fca-490b715ec38d nodeName:}" failed. No retries permitted until 2026-11-03 21:52:54.069036139 +0000 UTC m=+2288.562958001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/287dcaa3-959c-4d4c-9fca-490b715ec38d-maas-api-tls") pod "maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk" (UID: "287dcaa3-959c-4d4c-9fca-490b715ec38d") : secret "maas-api-e2e-praxis-a-ca758d-serving-cert" not found
Nov 03 21:52:53.569528 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:53.569500    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/287dcaa3-959c-4d4c-9fca-490b715ec38d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk\" (UID: \"287dcaa3-959c-4d4c-9fca-490b715ec38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"
Nov 03 21:52:53.571515 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:53.571496    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287dcaa3-959c-4d4c-9fca-490b715ec38d-metrics-certs\") pod \"maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk\" (UID: \"287dcaa3-959c-4d4c-9fca-490b715ec38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"
Nov 03 21:52:53.641316 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:53.641275    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45hq6\" (UniqueName: \"kubernetes.io/projected/287dcaa3-959c-4d4c-9fca-490b715ec38d-kube-api-access-45hq6\") pod \"maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk\" (UID: \"287dcaa3-959c-4d4c-9fca-490b715ec38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"
Nov 03 21:52:53.668456 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:53.668423    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e6dbdf0-e267-42b5-a916-4f39502de63d" path="/var/lib/kubelet/pods/1e6dbdf0-e267-42b5-a916-4f39502de63d/volumes"
Nov 03 21:52:54.074762 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:54.074725    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/287dcaa3-959c-4d4c-9fca-490b715ec38d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk\" (UID: \"287dcaa3-959c-4d4c-9fca-490b715ec38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"
Nov 03 21:52:54.077479 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:54.077458    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/287dcaa3-959c-4d4c-9fca-490b715ec38d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk\" (UID: \"287dcaa3-959c-4d4c-9fca-490b715ec38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"
Nov 03 21:52:54.184089 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:54.184046    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw" event={"ID":"dfb3e395-5cd3-47bb-b0d2-16cd8958bcae","Type":"ContainerStarted","Data":"bfa56daebeab5a1141372fcac9490bfeb98dbf61cab0514047e94d6d8aec08fd"}
Nov 03 21:52:54.184255 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:54.184114    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw" event={"ID":"dfb3e395-5cd3-47bb-b0d2-16cd8958bcae","Type":"ContainerStarted","Data":"10bb4249bbc345a2ce47a1499834829663a69831d54a3d64d17c566b5e398999"}
Nov 03 21:52:54.299470 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:54.299420    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw" podStartSLOduration=2.299405947 podStartE2EDuration="2.299405947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:52:54.296010558 +0000 UTC m=+2288.789932433" watchObservedRunningTime="2026-11-03 21:52:54.299405947 +0000 UTC m=+2288.793327821"
Nov 03 21:52:54.346303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:54.346103    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"
Nov 03 21:52:54.778124 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:54.778092    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"]
Nov 03 21:52:54.778964 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:52:54.778942    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod287dcaa3_959c_4d4c_9fca_490b715ec38d.slice/crio-4a82f5d5b99da72a5f617722b61b6d8b872db9a5b177727ce3c8085241443bdc WatchSource:0}: Error finding container 4a82f5d5b99da72a5f617722b61b6d8b872db9a5b177727ce3c8085241443bdc: Status 404 returned error can't find the container with id 4a82f5d5b99da72a5f617722b61b6d8b872db9a5b177727ce3c8085241443bdc
Nov 03 21:52:54.780900 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:54.780879    5053 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:52:55.191673 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:55.191581    5053 generic.go:356] "Generic (PLEG): container finished" podID="dfb3e395-5cd3-47bb-b0d2-16cd8958bcae" containerID="bfa56daebeab5a1141372fcac9490bfeb98dbf61cab0514047e94d6d8aec08fd" exitCode=0
Nov 03 21:52:55.191673 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:55.191662    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw" event={"ID":"dfb3e395-5cd3-47bb-b0d2-16cd8958bcae","Type":"ContainerDied","Data":"bfa56daebeab5a1141372fcac9490bfeb98dbf61cab0514047e94d6d8aec08fd"}
Nov 03 21:52:55.193003 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:55.192979    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk" event={"ID":"287dcaa3-959c-4d4c-9fca-490b715ec38d","Type":"ContainerStarted","Data":"4a82f5d5b99da72a5f617722b61b6d8b872db9a5b177727ce3c8085241443bdc"}
Nov 03 21:52:56.201457 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:56.201415    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk" event={"ID":"287dcaa3-959c-4d4c-9fca-490b715ec38d","Type":"ContainerStarted","Data":"430150a7e6ffd144535a7615568765ceaaad4bc9509a57773d132c93e2584b09"}
Nov 03 21:52:56.201862 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:56.201702    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"
Nov 03 21:52:56.322963 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:56.321866    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk" podStartSLOduration=2.408826789 podStartE2EDuration="3.32184699s" totalImagesPullingTime="913.020201ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:52:56.319604536 +0000 UTC m=+2290.813526411" watchObservedRunningTime="2026-11-03 21:52:56.32184699 +0000 UTC m=+2290.815768864"
Nov 03 21:52:56.366213 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:56.366187    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw"
Nov 03 21:52:56.404672 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:56.404636    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfb3e395-5cd3-47bb-b0d2-16cd8958bcae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfb3e395-5cd3-47bb-b0d2-16cd8958bcae-maas-api-openshift-ca\") pod \"dfb3e395-5cd3-47bb-b0d2-16cd8958bcae\" (UID: \"dfb3e395-5cd3-47bb-b0d2-16cd8958bcae\") "
Nov 03 21:52:56.404864 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:56.404839    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfb3e395-5cd3-47bb-b0d2-16cd8958bcae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfb3e395-5cd3-47bb-b0d2-16cd8958bcae-maas-api-xks-ca\") pod \"dfb3e395-5cd3-47bb-b0d2-16cd8958bcae\" (UID: \"dfb3e395-5cd3-47bb-b0d2-16cd8958bcae\") "
Nov 03 21:52:56.404950 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:56.404934    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb3e395-5cd3-47bb-b0d2-16cd8958bcae-kube-api-access-gjt5x\" (UniqueName: \"kubernetes.io/projected/dfb3e395-5cd3-47bb-b0d2-16cd8958bcae-kube-api-access-gjt5x\") pod \"dfb3e395-5cd3-47bb-b0d2-16cd8958bcae\" (UID: \"dfb3e395-5cd3-47bb-b0d2-16cd8958bcae\") "
Nov 03 21:52:56.405067 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:56.405042    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfb3e395-5cd3-47bb-b0d2-16cd8958bcae-maas-api-openshift-ca" pod "dfb3e395-5cd3-47bb-b0d2-16cd8958bcae" (UID: "dfb3e395-5cd3-47bb-b0d2-16cd8958bcae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:52:56.405577 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:56.405552    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfb3e395-5cd3-47bb-b0d2-16cd8958bcae-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:52:56.407332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:56.407308    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb3e395-5cd3-47bb-b0d2-16cd8958bcae-kube-api-access-gjt5x" pod "dfb3e395-5cd3-47bb-b0d2-16cd8958bcae" (UID: "dfb3e395-5cd3-47bb-b0d2-16cd8958bcae"). InnerVolumeSpecName "kube-api-access-gjt5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:52:56.407702 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:56.407679    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfb3e395-5cd3-47bb-b0d2-16cd8958bcae-maas-api-xks-ca" pod "dfb3e395-5cd3-47bb-b0d2-16cd8958bcae" (UID: "dfb3e395-5cd3-47bb-b0d2-16cd8958bcae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:52:56.507265 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:56.507150    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfb3e395-5cd3-47bb-b0d2-16cd8958bcae-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:52:56.507265 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:56.507192    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjt5x\" (UniqueName: \"kubernetes.io/projected/dfb3e395-5cd3-47bb-b0d2-16cd8958bcae-kube-api-access-gjt5x\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:52:57.209907 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:57.209868    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw"
Nov 03 21:52:57.209907 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:57.209887    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw" event={"ID":"dfb3e395-5cd3-47bb-b0d2-16cd8958bcae","Type":"ContainerDied","Data":"10bb4249bbc345a2ce47a1499834829663a69831d54a3d64d17c566b5e398999"}
Nov 03 21:52:57.210501 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:52:57.209936    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10bb4249bbc345a2ce47a1499834829663a69831d54a3d64d17c566b5e398999"
Nov 03 21:53:02.215165 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:02.215134    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"
Nov 03 21:53:02.998950 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:02.998911    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"]
Nov 03 21:53:02.999654 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:02.999633    5053 cpu_manager.go:414] "containerMap: removing container" podUID="dfb3e395-5cd3-47bb-b0d2-16cd8958bcae" containerName="revoke-keys"
Nov 03 21:53:02.999654 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:02.999654    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfb3e395-5cd3-47bb-b0d2-16cd8958bcae" containerName="revoke-keys"
Nov 03 21:53:02.999846 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:02.999821    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfb3e395-5cd3-47bb-b0d2-16cd8958bcae" containerName="revoke-keys"
Nov 03 21:53:03.003815 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.003792    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.105571 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.105537    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-c993b8-openshift-default-dockercfg-78kkt\""
Nov 03 21:53:03.111907 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.111578    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"]
Nov 03 21:53:03.116323 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.113540    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-podinfo\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.116323 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.113627    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-data\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.116323 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.113681    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-credential-socket\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.116323 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.113708    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-envoy\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.116323 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.113839    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjhjt\" (UniqueName: \"kubernetes.io/projected/ac387b47-5c73-4984-bda9-8b5c82af02a5-kube-api-access-kjhjt\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.116323 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.113880    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac387b47-5c73-4984-bda9-8b5c82af02a5-istiod-ca-cert\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.116323 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.113912    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-token\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.116323 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.114012    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-workload-certs\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.116323 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.114058    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-workload-socket\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.215568 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.215523    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-credential-socket\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.215568 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.215566    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-envoy\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.216080 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.215637    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjhjt\" (UniqueName: \"kubernetes.io/projected/ac387b47-5c73-4984-bda9-8b5c82af02a5-kube-api-access-kjhjt\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.216080 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.215661    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac387b47-5c73-4984-bda9-8b5c82af02a5-istiod-ca-cert\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.216080 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.215683    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-token\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.216080 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.215759    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-workload-certs\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.216080 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.215803    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-workload-socket\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.216080 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.215864    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-podinfo\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.216080 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.215931    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-data\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.216080 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.216013    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-credential-socket\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.216417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.216147    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-workload-certs\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.216417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.216279    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-workload-socket\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.216417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.216374    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-data\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.216567 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.216417    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac387b47-5c73-4984-bda9-8b5c82af02a5-istiod-ca-cert\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.218762 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.218739    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-envoy\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.218947 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.218923    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-podinfo\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.272621 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.272544    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-token\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.272908 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.272885    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjhjt\" (UniqueName: \"kubernetes.io/projected/ac387b47-5c73-4984-bda9-8b5c82af02a5-kube-api-access-kjhjt\") pod \"e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") " pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.318516 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.318479    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:03.757217 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.757192    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"]
Nov 03 21:53:03.757998 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:53:03.757974    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac387b47_5c73_4984_bda9_8b5c82af02a5.slice/crio-38c567c611120be8785c27d8900c1cb0fae7dd89033250f0c840ac1050be3d53 WatchSource:0}: Error finding container 38c567c611120be8785c27d8900c1cb0fae7dd89033250f0c840ac1050be3d53: Status 404 returned error can't find the container with id 38c567c611120be8785c27d8900c1cb0fae7dd89033250f0c840ac1050be3d53
Nov 03 21:53:03.760580 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.760541    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:53:03.760695 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.760631    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:53:03.760695 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:03.760681    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:53:04.258972 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:04.258937    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s" event={"ID":"ac387b47-5c73-4984-bda9-8b5c82af02a5","Type":"ContainerStarted","Data":"d6e43512cf8808bd2cb9564c7ec1505b947d81e3c8fccca628e536a11bf65372"}
Nov 03 21:53:04.259446 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:04.259002    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s" event={"ID":"ac387b47-5c73-4984-bda9-8b5c82af02a5","Type":"ContainerStarted","Data":"38c567c611120be8785c27d8900c1cb0fae7dd89033250f0c840ac1050be3d53"}
Nov 03 21:53:04.319658 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:04.319631    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:04.325643 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:04.325619    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:04.377993 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:04.377943    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s" podStartSLOduration=2.377929814 podStartE2EDuration="2.377929814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:53:04.376394801 +0000 UTC m=+2298.870316678" watchObservedRunningTime="2026-11-03 21:53:04.377929814 +0000 UTC m=+2298.871851688"
Nov 03 21:53:05.265858 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:05.265832    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:05.266854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:05.266833    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:53:10.931336 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:53:10.931300    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:53:22.179333 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:22.179297    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw"]
Nov 03 21:53:22.235339 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:22.235095    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3940f8-gcttw"]
Nov 03 21:53:23.668545 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:23.668463    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfb3e395-5cd3-47bb-b0d2-16cd8958bcae" path="/var/lib/kubelet/pods/dfb3e395-5cd3-47bb-b0d2-16cd8958bcae/volumes"
Nov 03 21:53:27.486865 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:27.486830    5053 generic.go:356] "Generic (PLEG): container finished" podID="d0510a33-96d0-4ad8-a5f3-3dba5a009fc7" containerID="9c350f8575b4e372536a24c13a1a58240f6d99afbfc8149d4f4ca0cabcff5078" exitCode=0
Nov 03 21:53:27.487339 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:27.486897    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4" event={"ID":"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7","Type":"ContainerDied","Data":"9c350f8575b4e372536a24c13a1a58240f6d99afbfc8149d4f4ca0cabcff5078"}
Nov 03 21:53:27.487339 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:27.487327    5053 scope.go:122] "RemoveContainer" containerID="9c350f8575b4e372536a24c13a1a58240f6d99afbfc8149d4f4ca0cabcff5078"
Nov 03 21:53:27.488829 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:27.488799    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:53:27.488893 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:27.488868    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:53:27.488930 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:27.488912    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:53:28.495525 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:28.495487    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4" event={"ID":"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7","Type":"ContainerStarted","Data":"6df2db91c842d2ff7b902e402008a66b8fe0829f36fcf10f87172b3317284d3a"}
Nov 03 21:53:28.857329 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:28.850492    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:53:28.857329 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:28.853086    5053 patch_prober.go:29] interesting pod/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 03 21:53:28.857329 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:28.853164    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4" podUID="d0510a33-96d0-4ad8-a5f3-3dba5a009fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 03 21:53:29.842983 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:29.842950    5053 patch_prober.go:29] interesting pod/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 03 21:53:29.843427 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:29.843005    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4" podUID="d0510a33-96d0-4ad8-a5f3-3dba5a009fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 03 21:53:30.842899 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:30.842868    5053 patch_prober.go:29] interesting pod/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 03 21:53:30.843084 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:30.842919    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4" podUID="d0510a33-96d0-4ad8-a5f3-3dba5a009fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 03 21:53:31.842935 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:31.842899    5053 patch_prober.go:29] interesting pod/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 03 21:53:31.843124 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:31.842953    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4" podUID="d0510a33-96d0-4ad8-a5f3-3dba5a009fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 03 21:53:32.842534 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:32.842504    5053 patch_prober.go:29] interesting pod/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 03 21:53:32.842725 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:32.842554    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4" podUID="d0510a33-96d0-4ad8-a5f3-3dba5a009fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 03 21:53:33.695186 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:33.695150    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:53:33.847985 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:33.847956    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:53:34.567570 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:34.567540    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:53:40.930501 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:53:40.930440    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:53:44.775800 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:44.775770    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t"]
Nov 03 21:53:44.780222 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:44.780203    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t"
Nov 03 21:53:44.853082 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:44.853050    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8079c204-d396-44db-9c96-aaef37273175-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t\" (UID: \"8079c204-d396-44db-9c96-aaef37273175\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t"
Nov 03 21:53:44.853331 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:44.853154    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwdp2\" (UniqueName: \"kubernetes.io/projected/8079c204-d396-44db-9c96-aaef37273175-kube-api-access-qwdp2\") pod \"maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t\" (UID: \"8079c204-d396-44db-9c96-aaef37273175\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t"
Nov 03 21:53:44.853331 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:44.853205    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8079c204-d396-44db-9c96-aaef37273175-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t\" (UID: \"8079c204-d396-44db-9c96-aaef37273175\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t"
Nov 03 21:53:44.853331 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:44.853228    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8079c204-d396-44db-9c96-aaef37273175-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t\" (UID: \"8079c204-d396-44db-9c96-aaef37273175\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t"
Nov 03 21:53:44.885992 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:44.885966    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t"]
Nov 03 21:53:44.954919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:44.954881    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8079c204-d396-44db-9c96-aaef37273175-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t\" (UID: \"8079c204-d396-44db-9c96-aaef37273175\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t"
Nov 03 21:53:44.955124 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:44.954945    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8079c204-d396-44db-9c96-aaef37273175-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t\" (UID: \"8079c204-d396-44db-9c96-aaef37273175\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t"
Nov 03 21:53:44.955350 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:44.955195    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8079c204-d396-44db-9c96-aaef37273175-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t\" (UID: \"8079c204-d396-44db-9c96-aaef37273175\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t"
Nov 03 21:53:44.955491 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:44.955352    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwdp2\" (UniqueName: \"kubernetes.io/projected/8079c204-d396-44db-9c96-aaef37273175-kube-api-access-qwdp2\") pod \"maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t\" (UID: \"8079c204-d396-44db-9c96-aaef37273175\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t"
Nov 03 21:53:44.956707 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:44.956590    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8079c204-d396-44db-9c96-aaef37273175-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t\" (UID: \"8079c204-d396-44db-9c96-aaef37273175\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t"
Nov 03 21:53:44.970058 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:44.970027    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8079c204-d396-44db-9c96-aaef37273175-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t\" (UID: \"8079c204-d396-44db-9c96-aaef37273175\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t"
Nov 03 21:53:44.970439 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:44.970418    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8079c204-d396-44db-9c96-aaef37273175-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t\" (UID: \"8079c204-d396-44db-9c96-aaef37273175\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t"
Nov 03 21:53:45.011837 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:45.011811    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwdp2\" (UniqueName: \"kubernetes.io/projected/8079c204-d396-44db-9c96-aaef37273175-kube-api-access-qwdp2\") pod \"maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t\" (UID: \"8079c204-d396-44db-9c96-aaef37273175\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t"
Nov 03 21:53:45.093620 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:45.093535    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t"
Nov 03 21:53:45.479572 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:45.479544    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t"]
Nov 03 21:53:45.480172 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:53:45.480147    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8079c204_d396_44db_9c96_aaef37273175.slice/crio-27d9a979f10bd0162eb384d5589e2616548959e2b9da52128bd9b70614f74410 WatchSource:0}: Error finding container 27d9a979f10bd0162eb384d5589e2616548959e2b9da52128bd9b70614f74410: Status 404 returned error can't find the container with id 27d9a979f10bd0162eb384d5589e2616548959e2b9da52128bd9b70614f74410
Nov 03 21:53:45.652598 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:45.652553    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t" event={"ID":"8079c204-d396-44db-9c96-aaef37273175","Type":"ContainerStarted","Data":"27d9a979f10bd0162eb384d5589e2616548959e2b9da52128bd9b70614f74410"}
Nov 03 21:53:46.662939 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:46.662841    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t" event={"ID":"8079c204-d396-44db-9c96-aaef37273175","Type":"ContainerStarted","Data":"6a0f479db2a9b35333e194d3ca508378ff832a5815a46756bb2d2f899160d2a5"}
Nov 03 21:53:46.662939 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:46.662894    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t"
Nov 03 21:53:46.777854 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:46.777798    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t" podStartSLOduration=1.977680246 podStartE2EDuration="2.777780628s" totalImagesPullingTime="800.100382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:53:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:53:46.776746173 +0000 UTC m=+2341.270668060" watchObservedRunningTime="2026-11-03 21:53:46.777780628 +0000 UTC m=+2341.271702505"
Nov 03 21:53:52.674339 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:52.674307    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t"
Nov 03 21:53:52.842336 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:52.842303    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"]
Nov 03 21:53:52.842585 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:52.842559    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c" podUID="84aa8472-f94e-4195-94e8-a83e9bc596d8" containerName="maas-api" containerID="cri-o://d9f5651d1e96fa32861019181a23eb33286ad9ac03e6ba69fe3ac862d2790e5d" gracePeriod=30
Nov 03 21:53:53.120248 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:53.120145    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"
Nov 03 21:53:53.163221 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:53.163187    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84aa8472-f94e-4195-94e8-a83e9bc596d8-kube-api-access-m9wqh\" (UniqueName: \"kubernetes.io/projected/84aa8472-f94e-4195-94e8-a83e9bc596d8-kube-api-access-m9wqh\") pod \"84aa8472-f94e-4195-94e8-a83e9bc596d8\" (UID: \"84aa8472-f94e-4195-94e8-a83e9bc596d8\") "
Nov 03 21:53:53.163452 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:53.163282    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84aa8472-f94e-4195-94e8-a83e9bc596d8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84aa8472-f94e-4195-94e8-a83e9bc596d8-maas-api-tls\") pod \"84aa8472-f94e-4195-94e8-a83e9bc596d8\" (UID: \"84aa8472-f94e-4195-94e8-a83e9bc596d8\") "
Nov 03 21:53:53.163452 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:53.163320    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84aa8472-f94e-4195-94e8-a83e9bc596d8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84aa8472-f94e-4195-94e8-a83e9bc596d8-openshift-service-ca\") pod \"84aa8472-f94e-4195-94e8-a83e9bc596d8\" (UID: \"84aa8472-f94e-4195-94e8-a83e9bc596d8\") "
Nov 03 21:53:53.163452 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:53.163361    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84aa8472-f94e-4195-94e8-a83e9bc596d8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/84aa8472-f94e-4195-94e8-a83e9bc596d8-metrics-certs\") pod \"84aa8472-f94e-4195-94e8-a83e9bc596d8\" (UID: \"84aa8472-f94e-4195-94e8-a83e9bc596d8\") "
Nov 03 21:53:53.163670 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:53.163642    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84aa8472-f94e-4195-94e8-a83e9bc596d8-openshift-service-ca" pod "84aa8472-f94e-4195-94e8-a83e9bc596d8" (UID: "84aa8472-f94e-4195-94e8-a83e9bc596d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:53:53.164272 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:53.164224    5053 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84aa8472-f94e-4195-94e8-a83e9bc596d8-openshift-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:53:53.165607 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:53.165580    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84aa8472-f94e-4195-94e8-a83e9bc596d8-metrics-certs" pod "84aa8472-f94e-4195-94e8-a83e9bc596d8" (UID: "84aa8472-f94e-4195-94e8-a83e9bc596d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:53:53.165701 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:53.165658    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84aa8472-f94e-4195-94e8-a83e9bc596d8-maas-api-tls" pod "84aa8472-f94e-4195-94e8-a83e9bc596d8" (UID: "84aa8472-f94e-4195-94e8-a83e9bc596d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:53:53.165701 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:53.165684    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84aa8472-f94e-4195-94e8-a83e9bc596d8-kube-api-access-m9wqh" pod "84aa8472-f94e-4195-94e8-a83e9bc596d8" (UID: "84aa8472-f94e-4195-94e8-a83e9bc596d8"). InnerVolumeSpecName "kube-api-access-m9wqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:53:53.266780 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:53.266693    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9wqh\" (UniqueName: \"kubernetes.io/projected/84aa8472-f94e-4195-94e8-a83e9bc596d8-kube-api-access-m9wqh\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:53:53.266780 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:53.266733    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84aa8472-f94e-4195-94e8-a83e9bc596d8-maas-api-tls\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:53:53.266780 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:53.266750    5053 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84aa8472-f94e-4195-94e8-a83e9bc596d8-metrics-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:53:53.708968 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:53.708929    5053 generic.go:356] "Generic (PLEG): container finished" podID="84aa8472-f94e-4195-94e8-a83e9bc596d8" containerID="d9f5651d1e96fa32861019181a23eb33286ad9ac03e6ba69fe3ac862d2790e5d" exitCode=0
Nov 03 21:53:53.709422 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:53.708993    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"
Nov 03 21:53:53.709422 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:53.709004    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c" event={"ID":"84aa8472-f94e-4195-94e8-a83e9bc596d8","Type":"ContainerDied","Data":"d9f5651d1e96fa32861019181a23eb33286ad9ac03e6ba69fe3ac862d2790e5d"}
Nov 03 21:53:53.709422 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:53.709061    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c" event={"ID":"84aa8472-f94e-4195-94e8-a83e9bc596d8","Type":"ContainerDied","Data":"3815cb140e88fcbe9cd332555225b2b2e096fef8450f15e323957a4578d628e9"}
Nov 03 21:53:53.709422 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:53.709082    5053 scope.go:122] "RemoveContainer" containerID="d9f5651d1e96fa32861019181a23eb33286ad9ac03e6ba69fe3ac862d2790e5d"
Nov 03 21:53:53.722154 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:53.722127    5053 scope.go:122] "RemoveContainer" containerID="d9f5651d1e96fa32861019181a23eb33286ad9ac03e6ba69fe3ac862d2790e5d"
Nov 03 21:53:53.722522 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:53:53.722500    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9f5651d1e96fa32861019181a23eb33286ad9ac03e6ba69fe3ac862d2790e5d\": container with ID starting with d9f5651d1e96fa32861019181a23eb33286ad9ac03e6ba69fe3ac862d2790e5d not found: ID does not exist" containerID="d9f5651d1e96fa32861019181a23eb33286ad9ac03e6ba69fe3ac862d2790e5d"
Nov 03 21:53:53.722602 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:53.722532    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9f5651d1e96fa32861019181a23eb33286ad9ac03e6ba69fe3ac862d2790e5d"} err="failed to get container status \"d9f5651d1e96fa32861019181a23eb33286ad9ac03e6ba69fe3ac862d2790e5d\": rpc error: code = NotFound desc = could not find container \"d9f5651d1e96fa32861019181a23eb33286ad9ac03e6ba69fe3ac862d2790e5d\": container with ID starting with d9f5651d1e96fa32861019181a23eb33286ad9ac03e6ba69fe3ac862d2790e5d not found: ID does not exist"
Nov 03 21:53:53.873333 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:53.873282    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"]
Nov 03 21:53:53.922676 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:53.922638    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-br77c"]
Nov 03 21:53:55.669417 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:53:55.669384    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84aa8472-f94e-4195-94e8-a83e9bc596d8" path="/var/lib/kubelet/pods/84aa8472-f94e-4195-94e8-a83e9bc596d8/volumes"
Nov 03 21:54:09.648991 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.648956    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"]
Nov 03 21:54:09.649911 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.649888    5053 cpu_manager.go:414] "containerMap: removing container" podUID="84aa8472-f94e-4195-94e8-a83e9bc596d8" containerName="maas-api"
Nov 03 21:54:09.649978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.649913    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84aa8472-f94e-4195-94e8-a83e9bc596d8" containerName="maas-api"
Nov 03 21:54:09.650051 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.650040    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="84aa8472-f94e-4195-94e8-a83e9bc596d8" containerName="maas-api"
Nov 03 21:54:09.654907 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.654884    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.753284 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.753205    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"]
Nov 03 21:54:09.756449 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.756420    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-292cf1-openshift-default-dockercfg-phdll\""
Nov 03 21:54:09.773174 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.773136    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af1b21a-e734-4f08-aab1-9d8308baeff6-istiod-ca-cert\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.773398 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.773189    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz6l5\" (UniqueName: \"kubernetes.io/projected/7af1b21a-e734-4f08-aab1-9d8308baeff6-kube-api-access-fz6l5\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.773398 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.773224    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-workload-socket\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.773398 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.773351    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-workload-certs\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.773608 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.773405    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-podinfo\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.773608 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.773566    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-data\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.773719 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.773604    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-envoy\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.773719 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.773647    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-credential-socket\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.773719 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.773684    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-token\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.895258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.877052    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af1b21a-e734-4f08-aab1-9d8308baeff6-istiod-ca-cert\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.895258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.877110    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz6l5\" (UniqueName: \"kubernetes.io/projected/7af1b21a-e734-4f08-aab1-9d8308baeff6-kube-api-access-fz6l5\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.895258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.877145    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-workload-socket\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.895258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.877221    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-workload-certs\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.895258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.877299    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-podinfo\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.895258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.877442    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-data\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.895258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.877478    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-envoy\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.895258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.877517    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-credential-socket\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.895258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.877555    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-token\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.895258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.881655    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-data\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.895258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.881923    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-workload-certs\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.895258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.882840    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-credential-socket\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.895258 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.886625    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af1b21a-e734-4f08-aab1-9d8308baeff6-istiod-ca-cert\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.923304 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.907574    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-workload-socket\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.923304 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.918910    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-podinfo\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.923304 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.921804    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-envoy\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.939200 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.939148    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz6l5\" (UniqueName: \"kubernetes.io/projected/7af1b21a-e734-4f08-aab1-9d8308baeff6-kube-api-access-fz6l5\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.942651 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.942625    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-token\") pod \"e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") " pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:09.970093 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:09.970063    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:10.396972 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:10.396924    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"]
Nov 03 21:54:10.436497 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:54:10.436470    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af1b21a_e734_4f08_aab1_9d8308baeff6.slice/crio-031a2c99d0ab7c7377d576ad143da6c44a63eac7b2941a3b87486762d14a36fe WatchSource:0}: Error finding container 031a2c99d0ab7c7377d576ad143da6c44a63eac7b2941a3b87486762d14a36fe: Status 404 returned error can't find the container with id 031a2c99d0ab7c7377d576ad143da6c44a63eac7b2941a3b87486762d14a36fe
Nov 03 21:54:10.438928 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:10.438890    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:54:10.439040 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:10.438972    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:54:10.439040 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:10.439017    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:54:10.928169 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:54:10.928134    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:54:10.947887 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:10.947853    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw" event={"ID":"7af1b21a-e734-4f08-aab1-9d8308baeff6","Type":"ContainerStarted","Data":"be6c84faf0201e0f5b797f5503eb86dfabfd31931aaa21a6cc7231efcd8cb702"}
Nov 03 21:54:10.948078 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:10.947914    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw" event={"ID":"7af1b21a-e734-4f08-aab1-9d8308baeff6","Type":"ContainerStarted","Data":"031a2c99d0ab7c7377d576ad143da6c44a63eac7b2941a3b87486762d14a36fe"}
Nov 03 21:54:10.970377 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:10.970353    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:11.074807 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:11.074753    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw" podStartSLOduration=2.074738 podStartE2EDuration="2.074738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:54:11.071449096 +0000 UTC m=+2365.565370970" watchObservedRunningTime="2026-11-03 21:54:11.074738 +0000 UTC m=+2365.568659930"
Nov 03 21:54:11.976375 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:11.976343    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:12.964303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:12.964271    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:12.965484 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:12.965460    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:54:40.929034 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:54:40.928993    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:54:46.056787 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:46.056758    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 21:54:46.062904 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:46.062878    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 21:54:46.066597 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:46.066576    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:54:46.071879 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:46.071861    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:54:48.638889 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.638856    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"]
Nov 03 21:54:48.645084 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.645054    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.762285 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.739090    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-credential-socket\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.762285 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.739165    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwgvg\" (UniqueName: \"kubernetes.io/projected/810ed10d-32b0-4e4f-ab41-37341380801a-kube-api-access-bwgvg\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.762285 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.739216    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-istio-data\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.762285 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.739282    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/810ed10d-32b0-4e4f-ab41-37341380801a-istio-token\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.762285 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.739321    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-workload-certs\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.762285 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.739386    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-workload-socket\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.762285 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.739519    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-istio-envoy\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.762285 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.739542    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/810ed10d-32b0-4e4f-ab41-37341380801a-istio-podinfo\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.762285 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.739621    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/810ed10d-32b0-4e4f-ab41-37341380801a-istiod-ca-cert\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.792205 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.792167    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-eb3520-openshift-default-dockercfg-jdx67\""
Nov 03 21:54:48.869402 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.869373    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/810ed10d-32b0-4e4f-ab41-37341380801a-istiod-ca-cert\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.869688 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.869671    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-credential-socket\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.870060 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.870047    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwgvg\" (UniqueName: \"kubernetes.io/projected/810ed10d-32b0-4e4f-ab41-37341380801a-kube-api-access-bwgvg\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.870174 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.870165    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-istio-data\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.870510 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.870498    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/810ed10d-32b0-4e4f-ab41-37341380801a-istio-token\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.870835 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.870821    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-workload-certs\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.871174 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.871162    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-workload-socket\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.871557 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.871546    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-istio-envoy\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.871936 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.871918    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/810ed10d-32b0-4e4f-ab41-37341380801a-istio-podinfo\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.872276 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.871103    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-workload-certs\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.872457 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.870457    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-istio-data\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.872582 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.871442    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-workload-socket\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.872704 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.870006    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-credential-socket\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.872790 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.870050    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/810ed10d-32b0-4e4f-ab41-37341380801a-istiod-ca-cert\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.880693 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.875097    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/810ed10d-32b0-4e4f-ab41-37341380801a-istio-podinfo\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.880693 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.878175    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"]
Nov 03 21:54:48.880693 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.880614    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-istio-envoy\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.927112 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.927028    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/810ed10d-32b0-4e4f-ab41-37341380801a-istio-token\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.927112 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.927083    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwgvg\" (UniqueName: \"kubernetes.io/projected/810ed10d-32b0-4e4f-ab41-37341380801a-kube-api-access-bwgvg\") pod \"e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") " pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:48.962322 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:48.962277    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:49.403049 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:49.402485    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"]
Nov 03 21:54:49.403049 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:54:49.402870    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810ed10d_32b0_4e4f_ab41_37341380801a.slice/crio-3c3f3347d56385ab8f682fe6f9c96e42fbca5d11bc97a63da31f4cac55bd9fab WatchSource:0}: Error finding container 3c3f3347d56385ab8f682fe6f9c96e42fbca5d11bc97a63da31f4cac55bd9fab: Status 404 returned error can't find the container with id 3c3f3347d56385ab8f682fe6f9c96e42fbca5d11bc97a63da31f4cac55bd9fab
Nov 03 21:54:49.407132 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:49.407088    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:54:49.407290 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:49.407176    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:54:49.407290 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:49.407219    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:54:50.314184 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:50.314143    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc" event={"ID":"810ed10d-32b0-4e4f-ab41-37341380801a","Type":"ContainerStarted","Data":"f1b8a8ce5d90f4c4473002c9ab8f121b30eebc99fe1d5d37d8c5892b913c2094"}
Nov 03 21:54:50.314655 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:50.314224    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc" event={"ID":"810ed10d-32b0-4e4f-ab41-37341380801a","Type":"ContainerStarted","Data":"3c3f3347d56385ab8f682fe6f9c96e42fbca5d11bc97a63da31f4cac55bd9fab"}
Nov 03 21:54:50.430331 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:50.430276    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc" podStartSLOduration=2.430257578 podStartE2EDuration="2.430257578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:54:50.428881669 +0000 UTC m=+2404.922803538" watchObservedRunningTime="2026-11-03 21:54:50.430257578 +0000 UTC m=+2404.924179455"
Nov 03 21:54:50.962911 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:50.962870    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:50.968503 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:50.968476    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:51.321680 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:51.321607    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:51.322707 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:51.322689    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:54:51.844620 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:51.844588    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t"]
Nov 03 21:54:51.844843 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:51.844813    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t" podUID="8079c204-d396-44db-9c96-aaef37273175" containerName="maas-api" containerID="cri-o://6a0f479db2a9b35333e194d3ca508378ff832a5815a46756bb2d2f899160d2a5" gracePeriod=30
Nov 03 21:54:52.140326 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:52.140209    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t"
Nov 03 21:54:52.214316 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:52.214285    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8079c204-d396-44db-9c96-aaef37273175-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8079c204-d396-44db-9c96-aaef37273175-maas-api-tls\") pod \"8079c204-d396-44db-9c96-aaef37273175\" (UID: \"8079c204-d396-44db-9c96-aaef37273175\") "
Nov 03 21:54:52.214501 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:52.214390    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8079c204-d396-44db-9c96-aaef37273175-kube-api-access-qwdp2\" (UniqueName: \"kubernetes.io/projected/8079c204-d396-44db-9c96-aaef37273175-kube-api-access-qwdp2\") pod \"8079c204-d396-44db-9c96-aaef37273175\" (UID: \"8079c204-d396-44db-9c96-aaef37273175\") "
Nov 03 21:54:52.214501 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:52.214438    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8079c204-d396-44db-9c96-aaef37273175-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8079c204-d396-44db-9c96-aaef37273175-openshift-service-ca\") pod \"8079c204-d396-44db-9c96-aaef37273175\" (UID: \"8079c204-d396-44db-9c96-aaef37273175\") "
Nov 03 21:54:52.214501 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:52.214475    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8079c204-d396-44db-9c96-aaef37273175-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8079c204-d396-44db-9c96-aaef37273175-metrics-certs\") pod \"8079c204-d396-44db-9c96-aaef37273175\" (UID: \"8079c204-d396-44db-9c96-aaef37273175\") "
Nov 03 21:54:52.214850 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:52.214823    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8079c204-d396-44db-9c96-aaef37273175-openshift-service-ca" pod "8079c204-d396-44db-9c96-aaef37273175" (UID: "8079c204-d396-44db-9c96-aaef37273175"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:54:52.215007 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:52.214984    5053 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8079c204-d396-44db-9c96-aaef37273175-openshift-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:54:52.216695 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:52.216670    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8079c204-d396-44db-9c96-aaef37273175-kube-api-access-qwdp2" pod "8079c204-d396-44db-9c96-aaef37273175" (UID: "8079c204-d396-44db-9c96-aaef37273175"). InnerVolumeSpecName "kube-api-access-qwdp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:54:52.217123 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:52.217105    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8079c204-d396-44db-9c96-aaef37273175-metrics-certs" pod "8079c204-d396-44db-9c96-aaef37273175" (UID: "8079c204-d396-44db-9c96-aaef37273175"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:54:52.217203 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:52.217124    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8079c204-d396-44db-9c96-aaef37273175-maas-api-tls" pod "8079c204-d396-44db-9c96-aaef37273175" (UID: "8079c204-d396-44db-9c96-aaef37273175"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:54:52.320637 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:52.320579    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8079c204-d396-44db-9c96-aaef37273175-maas-api-tls\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:54:52.320637 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:52.320624    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwdp2\" (UniqueName: \"kubernetes.io/projected/8079c204-d396-44db-9c96-aaef37273175-kube-api-access-qwdp2\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:54:52.320960 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:52.320647    5053 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8079c204-d396-44db-9c96-aaef37273175-metrics-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:54:52.338774 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:52.338734    5053 generic.go:356] "Generic (PLEG): container finished" podID="8079c204-d396-44db-9c96-aaef37273175" containerID="6a0f479db2a9b35333e194d3ca508378ff832a5815a46756bb2d2f899160d2a5" exitCode=0
Nov 03 21:54:52.339311 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:52.338776    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t" event={"ID":"8079c204-d396-44db-9c96-aaef37273175","Type":"ContainerDied","Data":"6a0f479db2a9b35333e194d3ca508378ff832a5815a46756bb2d2f899160d2a5"}
Nov 03 21:54:52.339311 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:52.338815    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t"
Nov 03 21:54:52.339311 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:52.338861    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t" event={"ID":"8079c204-d396-44db-9c96-aaef37273175","Type":"ContainerDied","Data":"27d9a979f10bd0162eb384d5589e2616548959e2b9da52128bd9b70614f74410"}
Nov 03 21:54:52.339311 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:52.338887    5053 scope.go:122] "RemoveContainer" containerID="6a0f479db2a9b35333e194d3ca508378ff832a5815a46756bb2d2f899160d2a5"
Nov 03 21:54:52.357921 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:52.357884    5053 scope.go:122] "RemoveContainer" containerID="6a0f479db2a9b35333e194d3ca508378ff832a5815a46756bb2d2f899160d2a5"
Nov 03 21:54:52.358303 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:54:52.358277    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a0f479db2a9b35333e194d3ca508378ff832a5815a46756bb2d2f899160d2a5\": container with ID starting with 6a0f479db2a9b35333e194d3ca508378ff832a5815a46756bb2d2f899160d2a5 not found: ID does not exist" containerID="6a0f479db2a9b35333e194d3ca508378ff832a5815a46756bb2d2f899160d2a5"
Nov 03 21:54:52.358381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:52.358314    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a0f479db2a9b35333e194d3ca508378ff832a5815a46756bb2d2f899160d2a5"} err="failed to get container status \"6a0f479db2a9b35333e194d3ca508378ff832a5815a46756bb2d2f899160d2a5\": rpc error: code = NotFound desc = could not find container \"6a0f479db2a9b35333e194d3ca508378ff832a5815a46756bb2d2f899160d2a5\": container with ID starting with 6a0f479db2a9b35333e194d3ca508378ff832a5815a46756bb2d2f899160d2a5 not found: ID does not exist"
Nov 03 21:54:52.520263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:52.516288    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t"]
Nov 03 21:54:52.522116 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:52.522085    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-jrv4t"]
Nov 03 21:54:53.668919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:54:53.668875    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8079c204-d396-44db-9c96-aaef37273175" path="/var/lib/kubelet/pods/8079c204-d396-44db-9c96-aaef37273175/volumes"
Nov 03 21:55:02.410988 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:02.410902    5053 generic.go:356] "Generic (PLEG): container finished" podID="ac387b47-5c73-4984-bda9-8b5c82af02a5" containerID="d6e43512cf8808bd2cb9564c7ec1505b947d81e3c8fccca628e536a11bf65372" exitCode=0
Nov 03 21:55:02.410988 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:02.410960    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s" event={"ID":"ac387b47-5c73-4984-bda9-8b5c82af02a5","Type":"ContainerDied","Data":"d6e43512cf8808bd2cb9564c7ec1505b947d81e3c8fccca628e536a11bf65372"}
Nov 03 21:55:02.411504 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:02.411397    5053 scope.go:122] "RemoveContainer" containerID="d6e43512cf8808bd2cb9564c7ec1505b947d81e3c8fccca628e536a11bf65372"
Nov 03 21:55:02.413006 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:02.412975    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:55:02.413075 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:02.413048    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:55:02.413111 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:02.413077    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:55:03.432850 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:03.432816    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s" event={"ID":"ac387b47-5c73-4984-bda9-8b5c82af02a5","Type":"ContainerStarted","Data":"68299c7f150ba045e58575e0ba430e41fe57525755a55899426289ee4192c0d7"}
Nov 03 21:55:04.319665 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:04.319629    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:55:04.326829 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:04.326421    5053 patch_prober.go:29] interesting pod/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Nov 03 21:55:04.326829 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:04.326496    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s" podUID="ac387b47-5c73-4984-bda9-8b5c82af02a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Nov 03 21:55:05.266099 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:05.266066    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:55:05.320262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:05.319587    5053 patch_prober.go:29] interesting pod/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Nov 03 21:55:05.320262 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:05.319646    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s" podUID="ac387b47-5c73-4984-bda9-8b5c82af02a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Nov 03 21:55:06.325750 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:06.325706    5053 patch_prober.go:29] interesting pod/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Nov 03 21:55:06.326174 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:06.325772    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s" podUID="ac387b47-5c73-4984-bda9-8b5c82af02a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Nov 03 21:55:07.325265 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:07.320665    5053 patch_prober.go:29] interesting pod/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Nov 03 21:55:07.325265 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:07.320727    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s" podUID="ac387b47-5c73-4984-bda9-8b5c82af02a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Nov 03 21:55:08.321826 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:08.321787    5053 patch_prober.go:29] interesting pod/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Nov 03 21:55:08.322510 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:08.322468    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s" podUID="ac387b47-5c73-4984-bda9-8b5c82af02a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Nov 03 21:55:09.324033 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:09.323994    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:55:09.325177 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:09.325159    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:55:10.928818 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:55:10.928763    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:55:14.799639 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:14.799607    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j"]
Nov 03 21:55:14.800272 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:14.800256    5053 cpu_manager.go:414] "containerMap: removing container" podUID="8079c204-d396-44db-9c96-aaef37273175" containerName="maas-api"
Nov 03 21:55:14.800272 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:14.800272    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8079c204-d396-44db-9c96-aaef37273175" containerName="maas-api"
Nov 03 21:55:14.800394 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:14.800375    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="8079c204-d396-44db-9c96-aaef37273175" containerName="maas-api"
Nov 03 21:55:14.802895 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:14.802876    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j"
Nov 03 21:55:14.906052 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:14.906013    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-292cf1-serving-cert\""
Nov 03 21:55:14.911302 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:14.911274    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j"]
Nov 03 21:55:14.977193 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:14.977160    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sggzb\" (UniqueName: \"kubernetes.io/projected/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-kube-api-access-sggzb\") pod \"maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j\" (UID: \"0c6a1be6-612c-46f0-ad39-07a4bc90d9a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j"
Nov 03 21:55:14.977374 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:14.977219    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j\" (UID: \"0c6a1be6-612c-46f0-ad39-07a4bc90d9a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j"
Nov 03 21:55:14.977420 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:14.977405    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-metrics-certs\") pod \"maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j\" (UID: \"0c6a1be6-612c-46f0-ad39-07a4bc90d9a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j"
Nov 03 21:55:14.977459 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:14.977446    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-maas-api-tls\") pod \"maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j\" (UID: \"0c6a1be6-612c-46f0-ad39-07a4bc90d9a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j"
Nov 03 21:55:15.082308 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:15.082201    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-metrics-certs\") pod \"maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j\" (UID: \"0c6a1be6-612c-46f0-ad39-07a4bc90d9a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j"
Nov 03 21:55:15.082308 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:15.082291    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-maas-api-tls\") pod \"maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j\" (UID: \"0c6a1be6-612c-46f0-ad39-07a4bc90d9a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j"
Nov 03 21:55:15.082521 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:15.082506    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sggzb\" (UniqueName: \"kubernetes.io/projected/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-kube-api-access-sggzb\") pod \"maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j\" (UID: \"0c6a1be6-612c-46f0-ad39-07a4bc90d9a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j"
Nov 03 21:55:15.082588 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:15.082554    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j\" (UID: \"0c6a1be6-612c-46f0-ad39-07a4bc90d9a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j"
Nov 03 21:55:15.083471 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:15.083445    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j\" (UID: \"0c6a1be6-612c-46f0-ad39-07a4bc90d9a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j"
Nov 03 21:55:15.087539 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:15.087510    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-maas-api-tls\") pod \"maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j\" (UID: \"0c6a1be6-612c-46f0-ad39-07a4bc90d9a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j"
Nov 03 21:55:15.088204 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:15.088177    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-metrics-certs\") pod \"maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j\" (UID: \"0c6a1be6-612c-46f0-ad39-07a4bc90d9a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j"
Nov 03 21:55:15.140173 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:15.140141    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sggzb\" (UniqueName: \"kubernetes.io/projected/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-kube-api-access-sggzb\") pod \"maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j\" (UID: \"0c6a1be6-612c-46f0-ad39-07a4bc90d9a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j"
Nov 03 21:55:15.419990 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:15.419908    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j"
Nov 03 21:55:15.794873 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:15.794844    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j"]
Nov 03 21:55:15.796039 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:55:15.796004    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6a1be6_612c_46f0_ad39_07a4bc90d9a1.slice/crio-7b3cb10d2675440728d3a201b1a05d7f112506ab3e65324edfb1fbefcb499b39 WatchSource:0}: Error finding container 7b3cb10d2675440728d3a201b1a05d7f112506ab3e65324edfb1fbefcb499b39: Status 404 returned error can't find the container with id 7b3cb10d2675440728d3a201b1a05d7f112506ab3e65324edfb1fbefcb499b39
Nov 03 21:55:16.541132 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:16.541094    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j" event={"ID":"0c6a1be6-612c-46f0-ad39-07a4bc90d9a1","Type":"ContainerStarted","Data":"7b3cb10d2675440728d3a201b1a05d7f112506ab3e65324edfb1fbefcb499b39"}
Nov 03 21:55:17.552189 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:17.552145    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j" event={"ID":"0c6a1be6-612c-46f0-ad39-07a4bc90d9a1","Type":"ContainerStarted","Data":"67b704561ea4e0c819aceabd2f5b667c93128a4e59b65d4a09927eacdff1d0c7"}
Nov 03 21:55:17.552708 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:17.552301    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j"
Nov 03 21:55:17.669328 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:17.669265    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j" podStartSLOduration=2.819201251 podStartE2EDuration="3.669231757s" totalImagesPullingTime="850.030506ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:55:17.666391788 +0000 UTC m=+2432.160313662" watchObservedRunningTime="2026-11-03 21:55:17.669231757 +0000 UTC m=+2432.163153631"
Nov 03 21:55:23.564421 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:23.564392    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j"
Nov 03 21:55:39.268893 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:39.268861    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"]
Nov 03 21:55:39.269458 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:39.269107    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95" podUID="48c6d17e-4e96-4837-8ec2-c1da7ecf30be" containerName="istio-proxy" containerID="cri-o://396905b992d761068cfea1a2f506667ffb86bb769580394cb61b8a2b052b73b2" gracePeriod=30
Nov 03 21:55:39.319525 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:39.319492    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"]
Nov 03 21:55:39.319785 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:39.319735    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5" podUID="469926c5-1bd5-415b-9c6a-54f4ff10f25a" containerName="istio-proxy" containerID="cri-o://d19f8a9ae0153d34cadf3c27e26b5afc6fd37bcad4c5665a98f3cb68a7cd4bb1" gracePeriod=30
Nov 03 21:55:40.964681 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:55:40.964625    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:55:41.244856 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:41.244762    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-blk95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:55:41.244856 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:41.244822    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95" podUID="48c6d17e-4e96-4837-8ec2-c1da7ecf30be" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:55:42.777415 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:42.777370    5053 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-jlbj5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:55:42.777920 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:42.777459    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5" podUID="469926c5-1bd5-415b-9c6a-54f4ff10f25a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:55:44.614800 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.614766    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:55:44.628754 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.628729    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-workload-socket\") pod \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") "
Nov 03 21:55:44.628986 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.628768    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-token\" (UniqueName: \"kubernetes.io/projected/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-token\") pod \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") "
Nov 03 21:55:44.628986 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.628811    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-kube-api-access-q95j7\" (UniqueName: \"kubernetes.io/projected/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-kube-api-access-q95j7\") pod \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") "
Nov 03 21:55:44.628986 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.628879    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istiod-ca-cert\") pod \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") "
Nov 03 21:55:44.628986 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.628976    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-data\") pod \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") "
Nov 03 21:55:44.629206 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.628982    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-workload-socket" pod "48c6d17e-4e96-4837-8ec2-c1da7ecf30be" (UID: "48c6d17e-4e96-4837-8ec2-c1da7ecf30be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:55:44.629206 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.629013    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-workload-certs\") pod \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") "
Nov 03 21:55:44.629206 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.629050    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-envoy\") pod \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") "
Nov 03 21:55:44.629206 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.629140    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-podinfo\") pod \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") "
Nov 03 21:55:44.629206 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.629188    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-credential-socket\") pod \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\" (UID: \"48c6d17e-4e96-4837-8ec2-c1da7ecf30be\") "
Nov 03 21:55:44.629485 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.629354    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istiod-ca-cert" pod "48c6d17e-4e96-4837-8ec2-c1da7ecf30be" (UID: "48c6d17e-4e96-4837-8ec2-c1da7ecf30be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:55:44.629874 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.629710    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-data" pod "48c6d17e-4e96-4837-8ec2-c1da7ecf30be" (UID: "48c6d17e-4e96-4837-8ec2-c1da7ecf30be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:55:44.629874 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.629828    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:55:44.629874 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.629843    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:55:44.629874 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.629852    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:55:44.630175 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.630117    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-credential-socket" pod "48c6d17e-4e96-4837-8ec2-c1da7ecf30be" (UID: "48c6d17e-4e96-4837-8ec2-c1da7ecf30be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:55:44.630402 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.630382    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-workload-certs" pod "48c6d17e-4e96-4837-8ec2-c1da7ecf30be" (UID: "48c6d17e-4e96-4837-8ec2-c1da7ecf30be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:55:44.631605 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.631584    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-envoy" pod "48c6d17e-4e96-4837-8ec2-c1da7ecf30be" (UID: "48c6d17e-4e96-4837-8ec2-c1da7ecf30be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:55:44.631916 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.631893    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-podinfo" pod "48c6d17e-4e96-4837-8ec2-c1da7ecf30be" (UID: "48c6d17e-4e96-4837-8ec2-c1da7ecf30be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:55:44.632572 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.632552    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-token" pod "48c6d17e-4e96-4837-8ec2-c1da7ecf30be" (UID: "48c6d17e-4e96-4837-8ec2-c1da7ecf30be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:55:44.633361 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.633342    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-kube-api-access-q95j7" pod "48c6d17e-4e96-4837-8ec2-c1da7ecf30be" (UID: "48c6d17e-4e96-4837-8ec2-c1da7ecf30be"). InnerVolumeSpecName "kube-api-access-q95j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:55:44.649714 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.649682    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:55:44.730343 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.730304    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-podinfo\") pod \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") "
Nov 03 21:55:44.730559 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.730381    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-workload-socket\") pod \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") "
Nov 03 21:55:44.730559 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.730439    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-token\" (UniqueName: \"kubernetes.io/projected/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-token\") pod \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") "
Nov 03 21:55:44.732250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.730603    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/469926c5-1bd5-415b-9c6a-54f4ff10f25a-kube-api-access-8v86d\" (UniqueName: \"kubernetes.io/projected/469926c5-1bd5-415b-9c6a-54f4ff10f25a-kube-api-access-8v86d\") pod \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") "
Nov 03 21:55:44.732250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.730708    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-workload-socket" pod "469926c5-1bd5-415b-9c6a-54f4ff10f25a" (UID: "469926c5-1bd5-415b-9c6a-54f4ff10f25a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:55:44.732250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.730750    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-credential-socket\") pod \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") "
Nov 03 21:55:44.732250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.730849    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-envoy\") pod \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") "
Nov 03 21:55:44.732250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.730928    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istiod-ca-cert\") pod \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") "
Nov 03 21:55:44.732250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.730931    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-credential-socket" pod "469926c5-1bd5-415b-9c6a-54f4ff10f25a" (UID: "469926c5-1bd5-415b-9c6a-54f4ff10f25a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:55:44.732250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.730978    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-workload-certs\") pod \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") "
Nov 03 21:55:44.732250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.731025    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-data\") pod \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\" (UID: \"469926c5-1bd5-415b-9c6a-54f4ff10f25a\") "
Nov 03 21:55:44.732250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.731313    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istiod-ca-cert" pod "469926c5-1bd5-415b-9c6a-54f4ff10f25a" (UID: "469926c5-1bd5-415b-9c6a-54f4ff10f25a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:55:44.732250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.731409    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-workload-certs" pod "469926c5-1bd5-415b-9c6a-54f4ff10f25a" (UID: "469926c5-1bd5-415b-9c6a-54f4ff10f25a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:55:44.732250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.731554    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-data" pod "469926c5-1bd5-415b-9c6a-54f4ff10f25a" (UID: "469926c5-1bd5-415b-9c6a-54f4ff10f25a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:55:44.732250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.731879    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:55:44.732250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.731898    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:55:44.732250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.731911    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:55:44.732250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.731923    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:55:44.732250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.731935    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q95j7\" (UniqueName: \"kubernetes.io/projected/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-kube-api-access-q95j7\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:55:44.732250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.731948    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:55:44.732250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.731961    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:55:44.732250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.731974    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c6d17e-4e96-4837-8ec2-c1da7ecf30be-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:55:44.732250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.731986    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:55:44.732250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.732000    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:55:44.732250 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.732012    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:55:44.734286 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.733559    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-podinfo" pod "469926c5-1bd5-415b-9c6a-54f4ff10f25a" (UID: "469926c5-1bd5-415b-9c6a-54f4ff10f25a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:55:44.734286 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.733594    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-token" pod "469926c5-1bd5-415b-9c6a-54f4ff10f25a" (UID: "469926c5-1bd5-415b-9c6a-54f4ff10f25a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:55:44.734286 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.733806    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-envoy" pod "469926c5-1bd5-415b-9c6a-54f4ff10f25a" (UID: "469926c5-1bd5-415b-9c6a-54f4ff10f25a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:55:44.735168 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.735147    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469926c5-1bd5-415b-9c6a-54f4ff10f25a-kube-api-access-8v86d" pod "469926c5-1bd5-415b-9c6a-54f4ff10f25a" (UID: "469926c5-1bd5-415b-9c6a-54f4ff10f25a"). InnerVolumeSpecName "kube-api-access-8v86d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:55:44.777540 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.777508    5053 generic.go:356] "Generic (PLEG): container finished" podID="469926c5-1bd5-415b-9c6a-54f4ff10f25a" containerID="d19f8a9ae0153d34cadf3c27e26b5afc6fd37bcad4c5665a98f3cb68a7cd4bb1" exitCode=0
Nov 03 21:55:44.777711 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.777555    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"
Nov 03 21:55:44.777711 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.777576    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5" event={"ID":"469926c5-1bd5-415b-9c6a-54f4ff10f25a","Type":"ContainerDied","Data":"d19f8a9ae0153d34cadf3c27e26b5afc6fd37bcad4c5665a98f3cb68a7cd4bb1"}
Nov 03 21:55:44.777711 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.777632    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5" event={"ID":"469926c5-1bd5-415b-9c6a-54f4ff10f25a","Type":"ContainerDied","Data":"cb5ebeb3d84f8799d7f2fe6e63f0000fb2c96cf0461f661413a6c9936d4a0b04"}
Nov 03 21:55:44.777711 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.777656    5053 scope.go:122] "RemoveContainer" containerID="d19f8a9ae0153d34cadf3c27e26b5afc6fd37bcad4c5665a98f3cb68a7cd4bb1"
Nov 03 21:55:44.779886 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.779838    5053 generic.go:356] "Generic (PLEG): container finished" podID="48c6d17e-4e96-4837-8ec2-c1da7ecf30be" containerID="396905b992d761068cfea1a2f506667ffb86bb769580394cb61b8a2b052b73b2" exitCode=0
Nov 03 21:55:44.779886 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.779867    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95" event={"ID":"48c6d17e-4e96-4837-8ec2-c1da7ecf30be","Type":"ContainerDied","Data":"396905b992d761068cfea1a2f506667ffb86bb769580394cb61b8a2b052b73b2"}
Nov 03 21:55:44.780055 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.779903    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"
Nov 03 21:55:44.780055 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.779938    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95" event={"ID":"48c6d17e-4e96-4837-8ec2-c1da7ecf30be","Type":"ContainerDied","Data":"bf4bd6d51e9ebdf363022a6884412fb60a191da27a3d5a343edd0c06d7397253"}
Nov 03 21:55:44.795556 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.795367    5053 scope.go:122] "RemoveContainer" containerID="d19f8a9ae0153d34cadf3c27e26b5afc6fd37bcad4c5665a98f3cb68a7cd4bb1"
Nov 03 21:55:44.799640 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:55:44.799413    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d19f8a9ae0153d34cadf3c27e26b5afc6fd37bcad4c5665a98f3cb68a7cd4bb1\": container with ID starting with d19f8a9ae0153d34cadf3c27e26b5afc6fd37bcad4c5665a98f3cb68a7cd4bb1 not found: ID does not exist" containerID="d19f8a9ae0153d34cadf3c27e26b5afc6fd37bcad4c5665a98f3cb68a7cd4bb1"
Nov 03 21:55:44.799640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.799475    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d19f8a9ae0153d34cadf3c27e26b5afc6fd37bcad4c5665a98f3cb68a7cd4bb1"} err="failed to get container status \"d19f8a9ae0153d34cadf3c27e26b5afc6fd37bcad4c5665a98f3cb68a7cd4bb1\": rpc error: code = NotFound desc = could not find container \"d19f8a9ae0153d34cadf3c27e26b5afc6fd37bcad4c5665a98f3cb68a7cd4bb1\": container with ID starting with d19f8a9ae0153d34cadf3c27e26b5afc6fd37bcad4c5665a98f3cb68a7cd4bb1 not found: ID does not exist"
Nov 03 21:55:44.799640 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.799504    5053 scope.go:122] "RemoveContainer" containerID="396905b992d761068cfea1a2f506667ffb86bb769580394cb61b8a2b052b73b2"
Nov 03 21:55:44.818833 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.818799    5053 scope.go:122] "RemoveContainer" containerID="7d7fb3b24a8708accbea047771ca706debce91a1360335ccf32223cc806171c7"
Nov 03 21:55:44.830929 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.830901    5053 scope.go:122] "RemoveContainer" containerID="396905b992d761068cfea1a2f506667ffb86bb769580394cb61b8a2b052b73b2"
Nov 03 21:55:44.831319 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:55:44.831290    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"396905b992d761068cfea1a2f506667ffb86bb769580394cb61b8a2b052b73b2\": container with ID starting with 396905b992d761068cfea1a2f506667ffb86bb769580394cb61b8a2b052b73b2 not found: ID does not exist" containerID="396905b992d761068cfea1a2f506667ffb86bb769580394cb61b8a2b052b73b2"
Nov 03 21:55:44.831440 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.831329    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396905b992d761068cfea1a2f506667ffb86bb769580394cb61b8a2b052b73b2"} err="failed to get container status \"396905b992d761068cfea1a2f506667ffb86bb769580394cb61b8a2b052b73b2\": rpc error: code = NotFound desc = could not find container \"396905b992d761068cfea1a2f506667ffb86bb769580394cb61b8a2b052b73b2\": container with ID starting with 396905b992d761068cfea1a2f506667ffb86bb769580394cb61b8a2b052b73b2 not found: ID does not exist"
Nov 03 21:55:44.831440 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.831355    5053 scope.go:122] "RemoveContainer" containerID="7d7fb3b24a8708accbea047771ca706debce91a1360335ccf32223cc806171c7"
Nov 03 21:55:44.831689 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:55:44.831668    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d7fb3b24a8708accbea047771ca706debce91a1360335ccf32223cc806171c7\": container with ID starting with 7d7fb3b24a8708accbea047771ca706debce91a1360335ccf32223cc806171c7 not found: ID does not exist" containerID="7d7fb3b24a8708accbea047771ca706debce91a1360335ccf32223cc806171c7"
Nov 03 21:55:44.831807 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.831695    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d7fb3b24a8708accbea047771ca706debce91a1360335ccf32223cc806171c7"} err="failed to get container status \"7d7fb3b24a8708accbea047771ca706debce91a1360335ccf32223cc806171c7\": rpc error: code = NotFound desc = could not find container \"7d7fb3b24a8708accbea047771ca706debce91a1360335ccf32223cc806171c7\": container with ID starting with 7d7fb3b24a8708accbea047771ca706debce91a1360335ccf32223cc806171c7 not found: ID does not exist"
Nov 03 21:55:44.833543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.833519    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:55:44.833637 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.833545    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:55:44.833637 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.833559    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469926c5-1bd5-415b-9c6a-54f4ff10f25a-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:55:44.833637 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.833571    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v86d\" (UniqueName: \"kubernetes.io/projected/469926c5-1bd5-415b-9c6a-54f4ff10f25a-kube-api-access-8v86d\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:55:44.999868 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:44.999830    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"]
Nov 03 21:55:45.002917 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:45.002893    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jlbj5"]
Nov 03 21:55:45.114660 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:45.114459    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"]
Nov 03 21:55:45.118812 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:45.118783    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-blk95"]
Nov 03 21:55:45.667696 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:45.667666    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="469926c5-1bd5-415b-9c6a-54f4ff10f25a" path="/var/lib/kubelet/pods/469926c5-1bd5-415b-9c6a-54f4ff10f25a/volumes"
Nov 03 21:55:45.668148 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:45.668134    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48c6d17e-4e96-4837-8ec2-c1da7ecf30be" path="/var/lib/kubelet/pods/48c6d17e-4e96-4837-8ec2-c1da7ecf30be/volumes"
Nov 03 21:55:47.827419 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:47.827323    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4"]
Nov 03 21:55:47.828339 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:47.828319    5053 cpu_manager.go:414] "containerMap: removing container" podUID="469926c5-1bd5-415b-9c6a-54f4ff10f25a" containerName="istio-proxy"
Nov 03 21:55:47.828411 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:47.828341    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="469926c5-1bd5-415b-9c6a-54f4ff10f25a" containerName="istio-proxy"
Nov 03 21:55:47.828411 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:47.828370    5053 cpu_manager.go:414] "containerMap: removing container" podUID="48c6d17e-4e96-4837-8ec2-c1da7ecf30be" containerName="istio-proxy"
Nov 03 21:55:47.828411 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:47.828377    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48c6d17e-4e96-4837-8ec2-c1da7ecf30be" containerName="istio-proxy"
Nov 03 21:55:47.828411 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:47.828393    5053 cpu_manager.go:414] "containerMap: removing container" podUID="48c6d17e-4e96-4837-8ec2-c1da7ecf30be" containerName="istio-proxy"
Nov 03 21:55:47.828411 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:47.828400    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48c6d17e-4e96-4837-8ec2-c1da7ecf30be" containerName="istio-proxy"
Nov 03 21:55:47.828569 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:47.828537    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="469926c5-1bd5-415b-9c6a-54f4ff10f25a" containerName="istio-proxy"
Nov 03 21:55:47.828569 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:47.828551    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="48c6d17e-4e96-4837-8ec2-c1da7ecf30be" containerName="istio-proxy"
Nov 03 21:55:47.828569 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:47.828561    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="48c6d17e-4e96-4837-8ec2-c1da7ecf30be" containerName="istio-proxy"
Nov 03 21:55:47.834573 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:47.834550    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4"
Nov 03 21:55:47.869789 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:47.869760    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c647dacd-0ef9-4c77-a255-d431abdf3b24-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4\" (UID: \"c647dacd-0ef9-4c77-a255-d431abdf3b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4"
Nov 03 21:55:47.869949 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:47.869841    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c647dacd-0ef9-4c77-a255-d431abdf3b24-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4\" (UID: \"c647dacd-0ef9-4c77-a255-d431abdf3b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4"
Nov 03 21:55:47.869949 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:47.869934    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c647dacd-0ef9-4c77-a255-d431abdf3b24-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4\" (UID: \"c647dacd-0ef9-4c77-a255-d431abdf3b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4"
Nov 03 21:55:47.870073 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:47.870003    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjfx7\" (UniqueName: \"kubernetes.io/projected/c647dacd-0ef9-4c77-a255-d431abdf3b24-kube-api-access-zjfx7\") pod \"maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4\" (UID: \"c647dacd-0ef9-4c77-a255-d431abdf3b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4"
Nov 03 21:55:47.935601 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:47.935572    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-6ddbba-serving-cert\""
Nov 03 21:55:47.941910 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:47.941877    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4"]
Nov 03 21:55:47.971589 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:47.971557    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c647dacd-0ef9-4c77-a255-d431abdf3b24-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4\" (UID: \"c647dacd-0ef9-4c77-a255-d431abdf3b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4"
Nov 03 21:55:47.971742 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:47.971594    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c647dacd-0ef9-4c77-a255-d431abdf3b24-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4\" (UID: \"c647dacd-0ef9-4c77-a255-d431abdf3b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4"
Nov 03 21:55:47.971742 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:47.971629    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjfx7\" (UniqueName: \"kubernetes.io/projected/c647dacd-0ef9-4c77-a255-d431abdf3b24-kube-api-access-zjfx7\") pod \"maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4\" (UID: \"c647dacd-0ef9-4c77-a255-d431abdf3b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4"
Nov 03 21:55:47.971883 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:47.971782    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c647dacd-0ef9-4c77-a255-d431abdf3b24-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4\" (UID: \"c647dacd-0ef9-4c77-a255-d431abdf3b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4"
Nov 03 21:55:47.972512 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:47.972438    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c647dacd-0ef9-4c77-a255-d431abdf3b24-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4\" (UID: \"c647dacd-0ef9-4c77-a255-d431abdf3b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4"
Nov 03 21:55:47.974278 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:47.974253    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c647dacd-0ef9-4c77-a255-d431abdf3b24-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4\" (UID: \"c647dacd-0ef9-4c77-a255-d431abdf3b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4"
Nov 03 21:55:47.974560 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:47.974539    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c647dacd-0ef9-4c77-a255-d431abdf3b24-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4\" (UID: \"c647dacd-0ef9-4c77-a255-d431abdf3b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4"
Nov 03 21:55:48.028106 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:48.028074    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjfx7\" (UniqueName: \"kubernetes.io/projected/c647dacd-0ef9-4c77-a255-d431abdf3b24-kube-api-access-zjfx7\") pod \"maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4\" (UID: \"c647dacd-0ef9-4c77-a255-d431abdf3b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4"
Nov 03 21:55:48.149710 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:48.149630    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4"
Nov 03 21:55:48.541172 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:48.541145    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4"]
Nov 03 21:55:48.543211 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:55:48.543184    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc647dacd_0ef9_4c77_a255_d431abdf3b24.slice/crio-86b7b5a55e37ec2fac1687d36ff51e4e3e8cbcd97477b6fad41b8eee002984cb WatchSource:0}: Error finding container 86b7b5a55e37ec2fac1687d36ff51e4e3e8cbcd97477b6fad41b8eee002984cb: Status 404 returned error can't find the container with id 86b7b5a55e37ec2fac1687d36ff51e4e3e8cbcd97477b6fad41b8eee002984cb
Nov 03 21:55:48.819319 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:48.817910    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4" event={"ID":"c647dacd-0ef9-4c77-a255-d431abdf3b24","Type":"ContainerStarted","Data":"86b7b5a55e37ec2fac1687d36ff51e4e3e8cbcd97477b6fad41b8eee002984cb"}
Nov 03 21:55:49.827169 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:49.827062    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4" event={"ID":"c647dacd-0ef9-4c77-a255-d431abdf3b24","Type":"ContainerStarted","Data":"d486cf3c4c771b54db222c8de0a02da8cbc660c6b447a8c615a8048d904c265c"}
Nov 03 21:55:49.827169 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:49.827136    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4"
Nov 03 21:55:49.944863 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:49.944797    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4" podStartSLOduration=2.030982467 podStartE2EDuration="2.944779857s" totalImagesPullingTime="913.79739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:55:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:55:49.9399745 +0000 UTC m=+2464.433896376" watchObservedRunningTime="2026-11-03 21:55:49.944779857 +0000 UTC m=+2464.438701733"
Nov 03 21:55:55.842585 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:55:55.842555    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4"
Nov 03 21:56:10.930749 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:56:10.930711    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:56:20.315607 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:20.315568    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"]
Nov 03 21:56:20.316407 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:20.315832    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb" podUID="fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" containerName="istio-proxy" containerID="cri-o://3aa569c93d320d5324a27563bc7934d73745191b1a58e35d6f98fe31d8621595" gracePeriod=30
Nov 03 21:56:22.362430 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:22.362394    5053 patch_prober.go:29] interesting pod/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:56:22.362827 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:22.362453    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc" podUID="810ed10d-32b0-4e4f-ab41-37341380801a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:56:24.092498 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:24.092421    5053 generic.go:356] "Generic (PLEG): container finished" podID="810ed10d-32b0-4e4f-ab41-37341380801a" containerID="f1b8a8ce5d90f4c4473002c9ab8f121b30eebc99fe1d5d37d8c5892b913c2094" exitCode=0
Nov 03 21:56:24.092498 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:24.092476    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc" event={"ID":"810ed10d-32b0-4e4f-ab41-37341380801a","Type":"ContainerDied","Data":"f1b8a8ce5d90f4c4473002c9ab8f121b30eebc99fe1d5d37d8c5892b913c2094"}
Nov 03 21:56:24.092933 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:24.092911    5053 scope.go:122] "RemoveContainer" containerID="f1b8a8ce5d90f4c4473002c9ab8f121b30eebc99fe1d5d37d8c5892b913c2094"
Nov 03 21:56:24.094434 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:24.094407    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:56:24.094489 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:24.094474    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:56:24.094530 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:24.094506    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:56:25.122259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:25.119075    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc" event={"ID":"810ed10d-32b0-4e4f-ab41-37341380801a","Type":"ContainerStarted","Data":"138b80baec640feb7fb8c9c3de91fef691a1831ee538346d146e24a423956eef"}
Nov 03 21:56:25.239261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:25.238547    5053 patch_prober.go:29] interesting pod/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:56:25.239261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:25.238626    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb" podUID="fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:56:25.963479 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:25.963442    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:56:25.965205 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:25.964994    5053 patch_prober.go:29] interesting pod/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Nov 03 21:56:25.965205 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:25.965045    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc" podUID="810ed10d-32b0-4e4f-ab41-37341380801a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Nov 03 21:56:26.136778 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.136703    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:56:26.140985 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.140956    5053 generic.go:356] "Generic (PLEG): container finished" podID="fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" containerID="3aa569c93d320d5324a27563bc7934d73745191b1a58e35d6f98fe31d8621595" exitCode=0
Nov 03 21:56:26.141365 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.141344    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"
Nov 03 21:56:26.141597 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.141347    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb" event={"ID":"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc","Type":"ContainerDied","Data":"3aa569c93d320d5324a27563bc7934d73745191b1a58e35d6f98fe31d8621595"}
Nov 03 21:56:26.141730 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.141715    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb" event={"ID":"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc","Type":"ContainerDied","Data":"bc1726b1e5b3d404e468e4157d07ed952692b61c6f621cb64c6e96835d5a487d"}
Nov 03 21:56:26.141848 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.141835    5053 scope.go:122] "RemoveContainer" containerID="3aa569c93d320d5324a27563bc7934d73745191b1a58e35d6f98fe31d8621595"
Nov 03 21:56:26.153524 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.153506    5053 scope.go:122] "RemoveContainer" containerID="009c614ee0742a684a3fa093be169b8d2fa2e56d364ae582be3c6742c4091432"
Nov 03 21:56:26.163371 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.163348    5053 scope.go:122] "RemoveContainer" containerID="3aa569c93d320d5324a27563bc7934d73745191b1a58e35d6f98fe31d8621595"
Nov 03 21:56:26.163688 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:56:26.163657    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aa569c93d320d5324a27563bc7934d73745191b1a58e35d6f98fe31d8621595\": container with ID starting with 3aa569c93d320d5324a27563bc7934d73745191b1a58e35d6f98fe31d8621595 not found: ID does not exist" containerID="3aa569c93d320d5324a27563bc7934d73745191b1a58e35d6f98fe31d8621595"
Nov 03 21:56:26.163796 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.163690    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa569c93d320d5324a27563bc7934d73745191b1a58e35d6f98fe31d8621595"} err="failed to get container status \"3aa569c93d320d5324a27563bc7934d73745191b1a58e35d6f98fe31d8621595\": rpc error: code = NotFound desc = could not find container \"3aa569c93d320d5324a27563bc7934d73745191b1a58e35d6f98fe31d8621595\": container with ID starting with 3aa569c93d320d5324a27563bc7934d73745191b1a58e35d6f98fe31d8621595 not found: ID does not exist"
Nov 03 21:56:26.163796 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.163708    5053 scope.go:122] "RemoveContainer" containerID="009c614ee0742a684a3fa093be169b8d2fa2e56d364ae582be3c6742c4091432"
Nov 03 21:56:26.164017 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:56:26.163998    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"009c614ee0742a684a3fa093be169b8d2fa2e56d364ae582be3c6742c4091432\": container with ID starting with 009c614ee0742a684a3fa093be169b8d2fa2e56d364ae582be3c6742c4091432 not found: ID does not exist" containerID="009c614ee0742a684a3fa093be169b8d2fa2e56d364ae582be3c6742c4091432"
Nov 03 21:56:26.164067 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.164023    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"009c614ee0742a684a3fa093be169b8d2fa2e56d364ae582be3c6742c4091432"} err="failed to get container status \"009c614ee0742a684a3fa093be169b8d2fa2e56d364ae582be3c6742c4091432\": rpc error: code = NotFound desc = could not find container \"009c614ee0742a684a3fa093be169b8d2fa2e56d364ae582be3c6742c4091432\": container with ID starting with 009c614ee0742a684a3fa093be169b8d2fa2e56d364ae582be3c6742c4091432 not found: ID does not exist"
Nov 03 21:56:26.195757 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.195731    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istiod-ca-cert\") pod \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") "
Nov 03 21:56:26.195879 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.195770    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-envoy\") pod \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") "
Nov 03 21:56:26.195879 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.195856    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-token\" (UniqueName: \"kubernetes.io/projected/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-token\") pod \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") "
Nov 03 21:56:26.195964 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.195885    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-credential-socket\") pod \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") "
Nov 03 21:56:26.195964 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.195943    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-workload-socket\") pod \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") "
Nov 03 21:56:26.196054 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.196031    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-kube-api-access-jbdcc\" (UniqueName: \"kubernetes.io/projected/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-kube-api-access-jbdcc\") pod \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") "
Nov 03 21:56:26.196113 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.196086    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istiod-ca-cert" pod "fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" (UID: "fc293ac5-37d8-4093-b8f4-5bdc2ba570bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:56:26.196176 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.196099    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-workload-certs\") pod \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") "
Nov 03 21:56:26.196176 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.196150    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-podinfo\") pod \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") "
Nov 03 21:56:26.196309 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.196194    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-credential-socket" pod "fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" (UID: "fc293ac5-37d8-4093-b8f4-5bdc2ba570bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:56:26.196309 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.196230    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-data\") pod \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\" (UID: \"fc293ac5-37d8-4093-b8f4-5bdc2ba570bc\") "
Nov 03 21:56:26.196410 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.196297    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-workload-socket" pod "fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" (UID: "fc293ac5-37d8-4093-b8f4-5bdc2ba570bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:56:26.196471 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.196410    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-workload-certs" pod "fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" (UID: "fc293ac5-37d8-4093-b8f4-5bdc2ba570bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:56:26.196615 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.196572    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-data" pod "fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" (UID: "fc293ac5-37d8-4093-b8f4-5bdc2ba570bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:56:26.197113 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.197088    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:56:26.197113 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.197119    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:56:26.197257 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.197135    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:56:26.197257 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.197149    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:56:26.197257 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.197160    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:56:26.198316 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.198288    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-envoy" pod "fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" (UID: "fc293ac5-37d8-4093-b8f4-5bdc2ba570bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:56:26.198601 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.198583    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-podinfo" pod "fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" (UID: "fc293ac5-37d8-4093-b8f4-5bdc2ba570bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:56:26.198829 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.198810    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-token" pod "fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" (UID: "fc293ac5-37d8-4093-b8f4-5bdc2ba570bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:56:26.198892 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.198832    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-kube-api-access-jbdcc" pod "fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" (UID: "fc293ac5-37d8-4093-b8f4-5bdc2ba570bc"). InnerVolumeSpecName "kube-api-access-jbdcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:56:26.298616 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.298577    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:56:26.298616 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.298617    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:56:26.298816 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.298633    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbdcc\" (UniqueName: \"kubernetes.io/projected/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-kube-api-access-jbdcc\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:56:26.298816 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.298648    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:56:26.571843 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.571810    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"]
Nov 03 21:56:26.574711 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.574683    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-1c3bed-openshift-default-7b4964d5cf-g5bbb"]
Nov 03 21:56:26.963576 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.963540    5053 patch_prober.go:29] interesting pod/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Nov 03 21:56:26.963771 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:26.963601    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc" podUID="810ed10d-32b0-4e4f-ab41-37341380801a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Nov 03 21:56:27.667288 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:27.667254    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" path="/var/lib/kubelet/pods/fc293ac5-37d8-4093-b8f4-5bdc2ba570bc/volumes"
Nov 03 21:56:27.962843 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:27.962811    5053 patch_prober.go:29] interesting pod/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Nov 03 21:56:27.963005 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:27.962862    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc" podUID="810ed10d-32b0-4e4f-ab41-37341380801a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Nov 03 21:56:28.962926 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:28.962894    5053 patch_prober.go:29] interesting pod/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Nov 03 21:56:28.963351 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:28.962953    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc" podUID="810ed10d-32b0-4e4f-ab41-37341380801a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Nov 03 21:56:29.962864 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:29.962828    5053 patch_prober.go:29] interesting pod/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Nov 03 21:56:29.963083 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:29.962898    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc" podUID="810ed10d-32b0-4e4f-ab41-37341380801a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Nov 03 21:56:30.967755 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:30.967724    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:56:30.968217 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:30.968142    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:56:30.968960 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:30.968941    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 21:56:39.283984 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:39.283954    5053 generic.go:356] "Generic (PLEG): container finished" podID="7af1b21a-e734-4f08-aab1-9d8308baeff6" containerID="be6c84faf0201e0f5b797f5503eb86dfabfd31931aaa21a6cc7231efcd8cb702" exitCode=0
Nov 03 21:56:39.284524 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:39.284029    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw" event={"ID":"7af1b21a-e734-4f08-aab1-9d8308baeff6","Type":"ContainerDied","Data":"be6c84faf0201e0f5b797f5503eb86dfabfd31931aaa21a6cc7231efcd8cb702"}
Nov 03 21:56:39.284612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:39.284598    5053 scope.go:122] "RemoveContainer" containerID="be6c84faf0201e0f5b797f5503eb86dfabfd31931aaa21a6cc7231efcd8cb702"
Nov 03 21:56:39.286178 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:39.286145    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:56:39.286308 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:39.286211    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:56:39.286308 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:39.286279    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:56:40.293375 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:40.293342    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw" event={"ID":"7af1b21a-e734-4f08-aab1-9d8308baeff6","Type":"ContainerStarted","Data":"91cf30917c1dfdec41ade6cd918f8e919cc948c523ef643c5ce20f65c8293e16"}
Nov 03 21:56:40.960186 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:56:40.960150    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:56:40.970552 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:40.970523    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:56:40.971646 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:40.971617    5053 patch_prober.go:29] interesting pod/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 03 21:56:40.971776 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:40.971669    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw" podUID="7af1b21a-e734-4f08-aab1-9d8308baeff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 03 21:56:41.971334 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:41.971300    5053 patch_prober.go:29] interesting pod/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 03 21:56:41.971735 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:41.971378    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw" podUID="7af1b21a-e734-4f08-aab1-9d8308baeff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 03 21:56:42.965171 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:42.965134    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:56:42.970966 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:42.970945    5053 patch_prober.go:29] interesting pod/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 03 21:56:42.971034 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:42.970994    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw" podUID="7af1b21a-e734-4f08-aab1-9d8308baeff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 03 21:56:43.970732 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:43.970699    5053 patch_prober.go:29] interesting pod/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 03 21:56:43.971143 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:43.970760    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw" podUID="7af1b21a-e734-4f08-aab1-9d8308baeff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 03 21:56:44.970518 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:44.970483    5053 patch_prober.go:29] interesting pod/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 03 21:56:44.970702 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:44.970551    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw" podUID="7af1b21a-e734-4f08-aab1-9d8308baeff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 03 21:56:45.974785 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:45.974750    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:56:45.975941 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:45.975909    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:56:46.121107 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:46.121070    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn"]
Nov 03 21:56:46.231393 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:46.231294    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx"]
Nov 03 21:56:51.121440 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:51.121398    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn" podUID="4ff958dd-42eb-4f29-88c4-f22e849bac96" containerName="payload-processing" containerID="cri-o://e8b4d1d871bc96984858d94142dc501afa43aa181bcc5c6c1c185af88211036f" gracePeriod=25
Nov 03 21:56:51.232704 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:51.232649    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx" podUID="5c79e9ce-b577-4034-8943-78ef2b0134e2" containerName="payload-pre-processing" containerID="cri-o://992eb0c4010421d86a8fa0630c0558c6d7e42a35418fc2e54390c81c7ae185be" gracePeriod=25
Nov 03 21:56:51.400447 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:51.400318    5053 generic.go:356] "Generic (PLEG): container finished" podID="5c79e9ce-b577-4034-8943-78ef2b0134e2" containerID="992eb0c4010421d86a8fa0630c0558c6d7e42a35418fc2e54390c81c7ae185be" exitCode=0
Nov 03 21:56:51.400620 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:51.400455    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx" event={"ID":"5c79e9ce-b577-4034-8943-78ef2b0134e2","Type":"ContainerDied","Data":"992eb0c4010421d86a8fa0630c0558c6d7e42a35418fc2e54390c81c7ae185be"}
Nov 03 21:56:51.402666 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:51.402644    5053 generic.go:356] "Generic (PLEG): container finished" podID="4ff958dd-42eb-4f29-88c4-f22e849bac96" containerID="e8b4d1d871bc96984858d94142dc501afa43aa181bcc5c6c1c185af88211036f" exitCode=0
Nov 03 21:56:51.402762 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:51.402721    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn" event={"ID":"4ff958dd-42eb-4f29-88c4-f22e849bac96","Type":"ContainerDied","Data":"e8b4d1d871bc96984858d94142dc501afa43aa181bcc5c6c1c185af88211036f"}
Nov 03 21:56:51.436522 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:51.436497    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn"
Nov 03 21:56:51.514134 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:51.514109    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx"
Nov 03 21:56:51.543798 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:51.543773    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c79e9ce-b577-4034-8943-78ef2b0134e2-kube-api-access-2dgkr\" (UniqueName: \"kubernetes.io/projected/5c79e9ce-b577-4034-8943-78ef2b0134e2-kube-api-access-2dgkr\") pod \"5c79e9ce-b577-4034-8943-78ef2b0134e2\" (UID: \"5c79e9ce-b577-4034-8943-78ef2b0134e2\") "
Nov 03 21:56:51.543962 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:51.543840    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c79e9ce-b577-4034-8943-78ef2b0134e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c79e9ce-b577-4034-8943-78ef2b0134e2-plugins-config-volume\") pod \"5c79e9ce-b577-4034-8943-78ef2b0134e2\" (UID: \"5c79e9ce-b577-4034-8943-78ef2b0134e2\") "
Nov 03 21:56:51.544045 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:51.544029    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ff958dd-42eb-4f29-88c4-f22e849bac96-kube-api-access-fk47m\" (UniqueName: \"kubernetes.io/projected/4ff958dd-42eb-4f29-88c4-f22e849bac96-kube-api-access-fk47m\") pod \"4ff958dd-42eb-4f29-88c4-f22e849bac96\" (UID: \"4ff958dd-42eb-4f29-88c4-f22e849bac96\") "
Nov 03 21:56:51.544127 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:51.544112    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ff958dd-42eb-4f29-88c4-f22e849bac96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ff958dd-42eb-4f29-88c4-f22e849bac96-plugins-config-volume\") pod \"4ff958dd-42eb-4f29-88c4-f22e849bac96\" (UID: \"4ff958dd-42eb-4f29-88c4-f22e849bac96\") "
Nov 03 21:56:51.544301 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:51.544270    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c79e9ce-b577-4034-8943-78ef2b0134e2-plugins-config-volume" pod "5c79e9ce-b577-4034-8943-78ef2b0134e2" (UID: "5c79e9ce-b577-4034-8943-78ef2b0134e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:56:51.544557 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:51.544534    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ff958dd-42eb-4f29-88c4-f22e849bac96-plugins-config-volume" pod "4ff958dd-42eb-4f29-88c4-f22e849bac96" (UID: "4ff958dd-42eb-4f29-88c4-f22e849bac96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:56:51.544830 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:51.544648    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ff958dd-42eb-4f29-88c4-f22e849bac96-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:56:51.544830 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:51.544666    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c79e9ce-b577-4034-8943-78ef2b0134e2-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:56:51.546225 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:51.546207    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ff958dd-42eb-4f29-88c4-f22e849bac96-kube-api-access-fk47m" pod "4ff958dd-42eb-4f29-88c4-f22e849bac96" (UID: "4ff958dd-42eb-4f29-88c4-f22e849bac96"). InnerVolumeSpecName "kube-api-access-fk47m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:56:51.546349 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:51.546288    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c79e9ce-b577-4034-8943-78ef2b0134e2-kube-api-access-2dgkr" pod "5c79e9ce-b577-4034-8943-78ef2b0134e2" (UID: "5c79e9ce-b577-4034-8943-78ef2b0134e2"). InnerVolumeSpecName "kube-api-access-2dgkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:56:51.646091 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:51.646057    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dgkr\" (UniqueName: \"kubernetes.io/projected/5c79e9ce-b577-4034-8943-78ef2b0134e2-kube-api-access-2dgkr\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:56:51.646091 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:51.646086    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk47m\" (UniqueName: \"kubernetes.io/projected/4ff958dd-42eb-4f29-88c4-f22e849bac96-kube-api-access-fk47m\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:56:52.411515 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:52.411412    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx"
Nov 03 21:56:52.411515 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:52.411421    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx" event={"ID":"5c79e9ce-b577-4034-8943-78ef2b0134e2","Type":"ContainerDied","Data":"28a80c71ac012bd8840e0e974ef381deb3e19eaa3d7be786837b6193b4272c5e"}
Nov 03 21:56:52.412064 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:52.411532    5053 scope.go:122] "RemoveContainer" containerID="992eb0c4010421d86a8fa0630c0558c6d7e42a35418fc2e54390c81c7ae185be"
Nov 03 21:56:52.413202 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:52.413181    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn"
Nov 03 21:56:52.413303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:52.413193    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn" event={"ID":"4ff958dd-42eb-4f29-88c4-f22e849bac96","Type":"ContainerDied","Data":"5170d67ec691d240137fc451d701de3e7cf2e37729dfbe9e68784ae0524f4660"}
Nov 03 21:56:52.424384 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:52.424361    5053 scope.go:122] "RemoveContainer" containerID="e8b4d1d871bc96984858d94142dc501afa43aa181bcc5c6c1c185af88211036f"
Nov 03 21:56:52.627947 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:52.627914    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx"]
Nov 03 21:56:52.677960 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:52.677886    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ca758d-77bbb67f6c-hlvgx"]
Nov 03 21:56:52.790067 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:52.790032    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn"]
Nov 03 21:56:52.793737 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:52.793712    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ca758d-5465c76ffb-8vztn"]
Nov 03 21:56:53.668879 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:53.668845    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ff958dd-42eb-4f29-88c4-f22e849bac96" path="/var/lib/kubelet/pods/4ff958dd-42eb-4f29-88c4-f22e849bac96/volumes"
Nov 03 21:56:53.669375 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:56:53.669348    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c79e9ce-b577-4034-8943-78ef2b0134e2" path="/var/lib/kubelet/pods/5c79e9ce-b577-4034-8943-78ef2b0134e2/volumes"
Nov 03 21:57:07.712660 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:07.712623    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4"]
Nov 03 21:57:07.713312 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:07.712876    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4" podUID="c647dacd-0ef9-4c77-a255-d431abdf3b24" containerName="maas-api" containerID="cri-o://d486cf3c4c771b54db222c8de0a02da8cbc660c6b447a8c615a8048d904c265c" gracePeriod=30
Nov 03 21:57:08.079529 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:08.079505    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4"
Nov 03 21:57:08.168903 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:08.168872    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c647dacd-0ef9-4c77-a255-d431abdf3b24-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c647dacd-0ef9-4c77-a255-d431abdf3b24-openshift-service-ca\") pod \"c647dacd-0ef9-4c77-a255-d431abdf3b24\" (UID: \"c647dacd-0ef9-4c77-a255-d431abdf3b24\") "
Nov 03 21:57:08.169135 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:08.168997    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c647dacd-0ef9-4c77-a255-d431abdf3b24-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c647dacd-0ef9-4c77-a255-d431abdf3b24-maas-api-tls\") pod \"c647dacd-0ef9-4c77-a255-d431abdf3b24\" (UID: \"c647dacd-0ef9-4c77-a255-d431abdf3b24\") "
Nov 03 21:57:08.169135 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:08.169043    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c647dacd-0ef9-4c77-a255-d431abdf3b24-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c647dacd-0ef9-4c77-a255-d431abdf3b24-metrics-certs\") pod \"c647dacd-0ef9-4c77-a255-d431abdf3b24\" (UID: \"c647dacd-0ef9-4c77-a255-d431abdf3b24\") "
Nov 03 21:57:08.169135 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:08.169088    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c647dacd-0ef9-4c77-a255-d431abdf3b24-kube-api-access-zjfx7\" (UniqueName: \"kubernetes.io/projected/c647dacd-0ef9-4c77-a255-d431abdf3b24-kube-api-access-zjfx7\") pod \"c647dacd-0ef9-4c77-a255-d431abdf3b24\" (UID: \"c647dacd-0ef9-4c77-a255-d431abdf3b24\") "
Nov 03 21:57:08.169371 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:08.169306    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c647dacd-0ef9-4c77-a255-d431abdf3b24-openshift-service-ca" pod "c647dacd-0ef9-4c77-a255-d431abdf3b24" (UID: "c647dacd-0ef9-4c77-a255-d431abdf3b24"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:57:08.169901 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:08.169773    5053 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c647dacd-0ef9-4c77-a255-d431abdf3b24-openshift-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:08.171436 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:08.171407    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c647dacd-0ef9-4c77-a255-d431abdf3b24-metrics-certs" pod "c647dacd-0ef9-4c77-a255-d431abdf3b24" (UID: "c647dacd-0ef9-4c77-a255-d431abdf3b24"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:57:08.171533 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:08.171459    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c647dacd-0ef9-4c77-a255-d431abdf3b24-kube-api-access-zjfx7" pod "c647dacd-0ef9-4c77-a255-d431abdf3b24" (UID: "c647dacd-0ef9-4c77-a255-d431abdf3b24"). InnerVolumeSpecName "kube-api-access-zjfx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:57:08.171533 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:08.171509    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c647dacd-0ef9-4c77-a255-d431abdf3b24-maas-api-tls" pod "c647dacd-0ef9-4c77-a255-d431abdf3b24" (UID: "c647dacd-0ef9-4c77-a255-d431abdf3b24"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:57:08.271230 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:08.271103    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c647dacd-0ef9-4c77-a255-d431abdf3b24-maas-api-tls\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:08.271230 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:08.271137    5053 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c647dacd-0ef9-4c77-a255-d431abdf3b24-metrics-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:08.271230 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:08.271154    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjfx7\" (UniqueName: \"kubernetes.io/projected/c647dacd-0ef9-4c77-a255-d431abdf3b24-kube-api-access-zjfx7\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:08.530670 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:08.530570    5053 generic.go:356] "Generic (PLEG): container finished" podID="c647dacd-0ef9-4c77-a255-d431abdf3b24" containerID="d486cf3c4c771b54db222c8de0a02da8cbc660c6b447a8c615a8048d904c265c" exitCode=0
Nov 03 21:57:08.530670 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:08.530630    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4"
Nov 03 21:57:08.530937 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:08.530707    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4" event={"ID":"c647dacd-0ef9-4c77-a255-d431abdf3b24","Type":"ContainerDied","Data":"d486cf3c4c771b54db222c8de0a02da8cbc660c6b447a8c615a8048d904c265c"}
Nov 03 21:57:08.530937 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:08.530747    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4" event={"ID":"c647dacd-0ef9-4c77-a255-d431abdf3b24","Type":"ContainerDied","Data":"86b7b5a55e37ec2fac1687d36ff51e4e3e8cbcd97477b6fad41b8eee002984cb"}
Nov 03 21:57:08.530937 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:08.530772    5053 scope.go:122] "RemoveContainer" containerID="d486cf3c4c771b54db222c8de0a02da8cbc660c6b447a8c615a8048d904c265c"
Nov 03 21:57:08.541976 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:08.541951    5053 scope.go:122] "RemoveContainer" containerID="d486cf3c4c771b54db222c8de0a02da8cbc660c6b447a8c615a8048d904c265c"
Nov 03 21:57:08.542315 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:57:08.542285    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d486cf3c4c771b54db222c8de0a02da8cbc660c6b447a8c615a8048d904c265c\": container with ID starting with d486cf3c4c771b54db222c8de0a02da8cbc660c6b447a8c615a8048d904c265c not found: ID does not exist" containerID="d486cf3c4c771b54db222c8de0a02da8cbc660c6b447a8c615a8048d904c265c"
Nov 03 21:57:08.542421 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:08.542322    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d486cf3c4c771b54db222c8de0a02da8cbc660c6b447a8c615a8048d904c265c"} err="failed to get container status \"d486cf3c4c771b54db222c8de0a02da8cbc660c6b447a8c615a8048d904c265c\": rpc error: code = NotFound desc = could not find container \"d486cf3c4c771b54db222c8de0a02da8cbc660c6b447a8c615a8048d904c265c\": container with ID starting with d486cf3c4c771b54db222c8de0a02da8cbc660c6b447a8c615a8048d904c265c not found: ID does not exist"
Nov 03 21:57:08.546711 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:57:08.546688    5053 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc647dacd_0ef9_4c77_a255_d431abdf3b24.slice/crio-86b7b5a55e37ec2fac1687d36ff51e4e3e8cbcd97477b6fad41b8eee002984cb\": RecentStats: unable to find data in memory cache]"
Nov 03 21:57:08.711694 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:08.706790    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4"]
Nov 03 21:57:08.756362 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:08.756327    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-5cfc5bb7cc-rjcm4"]
Nov 03 21:57:09.667414 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:09.667380    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c647dacd-0ef9-4c77-a255-d431abdf3b24" path="/var/lib/kubelet/pods/c647dacd-0ef9-4c77-a255-d431abdf3b24/volumes"
Nov 03 21:57:10.930156 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:57:10.930122    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:57:18.573869 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.573829    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"]
Nov 03 21:57:18.574493 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.574476    5053 cpu_manager.go:414] "containerMap: removing container" podUID="fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" containerName="istio-proxy"
Nov 03 21:57:18.574578 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.574497    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" containerName="istio-proxy"
Nov 03 21:57:18.574578 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.574515    5053 cpu_manager.go:414] "containerMap: removing container" podUID="5c79e9ce-b577-4034-8943-78ef2b0134e2" containerName="payload-pre-processing"
Nov 03 21:57:18.574578 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.574520    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c79e9ce-b577-4034-8943-78ef2b0134e2" containerName="payload-pre-processing"
Nov 03 21:57:18.574578 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.574556    5053 cpu_manager.go:414] "containerMap: removing container" podUID="c647dacd-0ef9-4c77-a255-d431abdf3b24" containerName="maas-api"
Nov 03 21:57:18.574578 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.574561    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c647dacd-0ef9-4c77-a255-d431abdf3b24" containerName="maas-api"
Nov 03 21:57:18.574578 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.574571    5053 cpu_manager.go:414] "containerMap: removing container" podUID="4ff958dd-42eb-4f29-88c4-f22e849bac96" containerName="payload-processing"
Nov 03 21:57:18.574578 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.574575    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ff958dd-42eb-4f29-88c4-f22e849bac96" containerName="payload-processing"
Nov 03 21:57:18.574804 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.574657    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" containerName="istio-proxy"
Nov 03 21:57:18.574804 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.574665    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c79e9ce-b577-4034-8943-78ef2b0134e2" containerName="payload-pre-processing"
Nov 03 21:57:18.574804 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.574675    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" containerName="istio-proxy"
Nov 03 21:57:18.574804 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.574684    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="c647dacd-0ef9-4c77-a255-d431abdf3b24" containerName="maas-api"
Nov 03 21:57:18.574804 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.574689    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ff958dd-42eb-4f29-88c4-f22e849bac96" containerName="payload-processing"
Nov 03 21:57:18.578445 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.578428    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.630076 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.630046    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-95d604-openshift-default-dockercfg-9zncw\""
Nov 03 21:57:18.683599 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.683562    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"]
Nov 03 21:57:18.712421 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.712378    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-workload-socket\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.712604 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.712452    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-credential-socket\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.712604 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.712524    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-istio-data\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.712604 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.712565    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/122b09a6-7732-417d-b82f-a977510c180f-istio-token\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.712803 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.712628    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/122b09a6-7732-417d-b82f-a977510c180f-istio-podinfo\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.712803 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.712668    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-istio-envoy\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.712803 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.712704    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-workload-certs\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.712803 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.712798    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/122b09a6-7732-417d-b82f-a977510c180f-istiod-ca-cert\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.720063 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.713680    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nw54\" (UniqueName: \"kubernetes.io/projected/122b09a6-7732-417d-b82f-a977510c180f-kube-api-access-5nw54\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.815622 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.815575    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/122b09a6-7732-417d-b82f-a977510c180f-istio-token\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.815835 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.815675    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/122b09a6-7732-417d-b82f-a977510c180f-istio-podinfo\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.815835 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.815712    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-istio-envoy\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.815835 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.815747    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-workload-certs\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.816004 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.815836    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/122b09a6-7732-417d-b82f-a977510c180f-istiod-ca-cert\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.816004 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.815882    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nw54\" (UniqueName: \"kubernetes.io/projected/122b09a6-7732-417d-b82f-a977510c180f-kube-api-access-5nw54\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.816004 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.815986    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-workload-socket\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.816144 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.816028    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-credential-socket\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.816144 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.816091    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-istio-data\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.816315 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.816225    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-workload-certs\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.816496 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.816466    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-istio-data\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.816570 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.816501    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-workload-socket\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.816625 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.816587    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-credential-socket\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.816625 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.816603    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/122b09a6-7732-417d-b82f-a977510c180f-istiod-ca-cert\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.818809 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.818772    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/122b09a6-7732-417d-b82f-a977510c180f-istio-podinfo\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.819000 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.818976    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-istio-envoy\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.876108 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.876028    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nw54\" (UniqueName: \"kubernetes.io/projected/122b09a6-7732-417d-b82f-a977510c180f-kube-api-access-5nw54\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.880652 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.880623    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/122b09a6-7732-417d-b82f-a977510c180f-istio-token\") pod \"e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") " pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:18.892656 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:18.892630    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:19.325247 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:19.325202    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"]
Nov 03 21:57:19.326096 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:57:19.326072    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122b09a6_7732_417d_b82f_a977510c180f.slice/crio-c0914d56429506ab2a1255f5da2aeba8828a23a6e170000dcd25a4312fc6bfba WatchSource:0}: Error finding container c0914d56429506ab2a1255f5da2aeba8828a23a6e170000dcd25a4312fc6bfba: Status 404 returned error can't find the container with id c0914d56429506ab2a1255f5da2aeba8828a23a6e170000dcd25a4312fc6bfba
Nov 03 21:57:19.328695 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:19.328661    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:57:19.328819 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:19.328739    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:57:19.328819 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:19.328783    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:57:19.608219 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:19.608121    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd" event={"ID":"122b09a6-7732-417d-b82f-a977510c180f","Type":"ContainerStarted","Data":"292948e4406f710ebace8840ca6143130122c4774da6958deb001b408cc68ebe"}
Nov 03 21:57:19.608219 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:19.608191    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd" event={"ID":"122b09a6-7732-417d-b82f-a977510c180f","Type":"ContainerStarted","Data":"c0914d56429506ab2a1255f5da2aeba8828a23a6e170000dcd25a4312fc6bfba"}
Nov 03 21:57:19.730377 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:19.730319    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd" podStartSLOduration=1.730300834 podStartE2EDuration="1.730300834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:57:19.72664923 +0000 UTC m=+2554.220571120" watchObservedRunningTime="2026-11-03 21:57:19.730300834 +0000 UTC m=+2554.224222712"
Nov 03 21:57:19.893056 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:19.892971    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:20.234767 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:20.234729    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"]
Nov 03 21:57:20.234965 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:20.234940    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk" podUID="287dcaa3-959c-4d4c-9fca-490b715ec38d" containerName="maas-api" containerID="cri-o://430150a7e6ffd144535a7615568765ceaaad4bc9509a57773d132c93e2584b09" gracePeriod=30
Nov 03 21:57:20.524551 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:20.524525    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"
Nov 03 21:57:20.653309 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:20.643535    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/287dcaa3-959c-4d4c-9fca-490b715ec38d-kube-api-access-45hq6\" (UniqueName: \"kubernetes.io/projected/287dcaa3-959c-4d4c-9fca-490b715ec38d-kube-api-access-45hq6\") pod \"287dcaa3-959c-4d4c-9fca-490b715ec38d\" (UID: \"287dcaa3-959c-4d4c-9fca-490b715ec38d\") "
Nov 03 21:57:20.653309 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:20.643598    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/287dcaa3-959c-4d4c-9fca-490b715ec38d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/287dcaa3-959c-4d4c-9fca-490b715ec38d-metrics-certs\") pod \"287dcaa3-959c-4d4c-9fca-490b715ec38d\" (UID: \"287dcaa3-959c-4d4c-9fca-490b715ec38d\") "
Nov 03 21:57:20.653309 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:20.643767    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/287dcaa3-959c-4d4c-9fca-490b715ec38d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/287dcaa3-959c-4d4c-9fca-490b715ec38d-maas-api-tls\") pod \"287dcaa3-959c-4d4c-9fca-490b715ec38d\" (UID: \"287dcaa3-959c-4d4c-9fca-490b715ec38d\") "
Nov 03 21:57:20.653309 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:20.643881    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/287dcaa3-959c-4d4c-9fca-490b715ec38d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/287dcaa3-959c-4d4c-9fca-490b715ec38d-openshift-service-ca\") pod \"287dcaa3-959c-4d4c-9fca-490b715ec38d\" (UID: \"287dcaa3-959c-4d4c-9fca-490b715ec38d\") "
Nov 03 21:57:20.653309 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:20.645028    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/287dcaa3-959c-4d4c-9fca-490b715ec38d-openshift-service-ca" pod "287dcaa3-959c-4d4c-9fca-490b715ec38d" (UID: "287dcaa3-959c-4d4c-9fca-490b715ec38d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:57:20.664538 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:20.657601    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/287dcaa3-959c-4d4c-9fca-490b715ec38d-kube-api-access-45hq6" pod "287dcaa3-959c-4d4c-9fca-490b715ec38d" (UID: "287dcaa3-959c-4d4c-9fca-490b715ec38d"). InnerVolumeSpecName "kube-api-access-45hq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:57:20.678190 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:20.678154    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/287dcaa3-959c-4d4c-9fca-490b715ec38d-metrics-certs" pod "287dcaa3-959c-4d4c-9fca-490b715ec38d" (UID: "287dcaa3-959c-4d4c-9fca-490b715ec38d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:57:20.683694 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:20.683662    5053 generic.go:356] "Generic (PLEG): container finished" podID="287dcaa3-959c-4d4c-9fca-490b715ec38d" containerID="430150a7e6ffd144535a7615568765ceaaad4bc9509a57773d132c93e2584b09" exitCode=0
Nov 03 21:57:20.683864 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:20.683759    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"
Nov 03 21:57:20.683930 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:20.683884    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk" event={"ID":"287dcaa3-959c-4d4c-9fca-490b715ec38d","Type":"ContainerDied","Data":"430150a7e6ffd144535a7615568765ceaaad4bc9509a57773d132c93e2584b09"}
Nov 03 21:57:20.683988 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:20.683935    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk" event={"ID":"287dcaa3-959c-4d4c-9fca-490b715ec38d","Type":"ContainerDied","Data":"4a82f5d5b99da72a5f617722b61b6d8b872db9a5b177727ce3c8085241443bdc"}
Nov 03 21:57:20.683988 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:20.683966    5053 scope.go:122] "RemoveContainer" containerID="430150a7e6ffd144535a7615568765ceaaad4bc9509a57773d132c93e2584b09"
Nov 03 21:57:20.685805 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:20.685704    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/287dcaa3-959c-4d4c-9fca-490b715ec38d-maas-api-tls" pod "287dcaa3-959c-4d4c-9fca-490b715ec38d" (UID: "287dcaa3-959c-4d4c-9fca-490b715ec38d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:57:20.694825 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:20.694800    5053 scope.go:122] "RemoveContainer" containerID="430150a7e6ffd144535a7615568765ceaaad4bc9509a57773d132c93e2584b09"
Nov 03 21:57:20.695118 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:57:20.695096    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"430150a7e6ffd144535a7615568765ceaaad4bc9509a57773d132c93e2584b09\": container with ID starting with 430150a7e6ffd144535a7615568765ceaaad4bc9509a57773d132c93e2584b09 not found: ID does not exist" containerID="430150a7e6ffd144535a7615568765ceaaad4bc9509a57773d132c93e2584b09"
Nov 03 21:57:20.695217 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:20.695126    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"430150a7e6ffd144535a7615568765ceaaad4bc9509a57773d132c93e2584b09"} err="failed to get container status \"430150a7e6ffd144535a7615568765ceaaad4bc9509a57773d132c93e2584b09\": rpc error: code = NotFound desc = could not find container \"430150a7e6ffd144535a7615568765ceaaad4bc9509a57773d132c93e2584b09\": container with ID starting with 430150a7e6ffd144535a7615568765ceaaad4bc9509a57773d132c93e2584b09 not found: ID does not exist"
Nov 03 21:57:20.745774 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:20.745739    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/287dcaa3-959c-4d4c-9fca-490b715ec38d-maas-api-tls\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:20.745774 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:20.745778    5053 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/287dcaa3-959c-4d4c-9fca-490b715ec38d-openshift-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:20.745992 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:20.745796    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45hq6\" (UniqueName: \"kubernetes.io/projected/287dcaa3-959c-4d4c-9fca-490b715ec38d-kube-api-access-45hq6\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:20.745992 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:20.745813    5053 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287dcaa3-959c-4d4c-9fca-490b715ec38d-metrics-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:20.899567 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:20.899538    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:21.125872 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:21.125791    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"]
Nov 03 21:57:21.130664 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:21.130638    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca758d-6f65dc5df7-4blhk"]
Nov 03 21:57:21.667420 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:21.667390    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="287dcaa3-959c-4d4c-9fca-490b715ec38d" path="/var/lib/kubelet/pods/287dcaa3-959c-4d4c-9fca-490b715ec38d/volumes"
Nov 03 21:57:21.692298 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:21.692267    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:21.693196 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:21.693180    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:57:23.725208 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:23.725177    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b2a8f46-bfh7k"]
Nov 03 21:57:23.725829 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:23.725812    5053 cpu_manager.go:414] "containerMap: removing container" podUID="287dcaa3-959c-4d4c-9fca-490b715ec38d" containerName="maas-api"
Nov 03 21:57:23.725893 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:23.725830    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="287dcaa3-959c-4d4c-9fca-490b715ec38d" containerName="maas-api"
Nov 03 21:57:23.725893 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:23.725849    5053 cpu_manager.go:414] "containerMap: removing container" podUID="fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" containerName="istio-proxy"
Nov 03 21:57:23.725893 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:23.725856    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc293ac5-37d8-4093-b8f4-5bdc2ba570bc" containerName="istio-proxy"
Nov 03 21:57:23.726045 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:23.725969    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="287dcaa3-959c-4d4c-9fca-490b715ec38d" containerName="maas-api"
Nov 03 21:57:23.730871 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:23.730848    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2a8f46-bfh7k"
Nov 03 21:57:23.785164 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:23.785132    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b2a8f46-bfh7k"]
Nov 03 21:57:23.788192 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:23.788169    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:57:23.788450 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:23.788427    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:57:23.885564 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:23.885527    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8n4m\" (UniqueName: \"kubernetes.io/projected/018a254b-8a68-402c-a0d2-ce1fbf23796f-kube-api-access-m8n4m\") pod \"maas-api-revoke-sub-0b2a8f46-bfh7k\" (UID: \"018a254b-8a68-402c-a0d2-ce1fbf23796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2a8f46-bfh7k"
Nov 03 21:57:23.885751 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:23.885589    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/018a254b-8a68-402c-a0d2-ce1fbf23796f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b2a8f46-bfh7k\" (UID: \"018a254b-8a68-402c-a0d2-ce1fbf23796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2a8f46-bfh7k"
Nov 03 21:57:23.885915 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:23.885895    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/018a254b-8a68-402c-a0d2-ce1fbf23796f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b2a8f46-bfh7k\" (UID: \"018a254b-8a68-402c-a0d2-ce1fbf23796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2a8f46-bfh7k"
Nov 03 21:57:23.986952 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:23.986862    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/018a254b-8a68-402c-a0d2-ce1fbf23796f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b2a8f46-bfh7k\" (UID: \"018a254b-8a68-402c-a0d2-ce1fbf23796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2a8f46-bfh7k"
Nov 03 21:57:23.986952 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:23.986927    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8n4m\" (UniqueName: \"kubernetes.io/projected/018a254b-8a68-402c-a0d2-ce1fbf23796f-kube-api-access-m8n4m\") pod \"maas-api-revoke-sub-0b2a8f46-bfh7k\" (UID: \"018a254b-8a68-402c-a0d2-ce1fbf23796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2a8f46-bfh7k"
Nov 03 21:57:23.987208 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:23.986956    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/018a254b-8a68-402c-a0d2-ce1fbf23796f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b2a8f46-bfh7k\" (UID: \"018a254b-8a68-402c-a0d2-ce1fbf23796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2a8f46-bfh7k"
Nov 03 21:57:23.987550 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:23.987525    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/018a254b-8a68-402c-a0d2-ce1fbf23796f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b2a8f46-bfh7k\" (UID: \"018a254b-8a68-402c-a0d2-ce1fbf23796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2a8f46-bfh7k"
Nov 03 21:57:23.989545 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:23.989526    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/018a254b-8a68-402c-a0d2-ce1fbf23796f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b2a8f46-bfh7k\" (UID: \"018a254b-8a68-402c-a0d2-ce1fbf23796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2a8f46-bfh7k"
Nov 03 21:57:24.044191 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:24.044153    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8n4m\" (UniqueName: \"kubernetes.io/projected/018a254b-8a68-402c-a0d2-ce1fbf23796f-kube-api-access-m8n4m\") pod \"maas-api-revoke-sub-0b2a8f46-bfh7k\" (UID: \"018a254b-8a68-402c-a0d2-ce1fbf23796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2a8f46-bfh7k"
Nov 03 21:57:24.044979 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:24.044962    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2a8f46-bfh7k"
Nov 03 21:57:24.440944 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:57:24.440915    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018a254b_8a68_402c_a0d2_ce1fbf23796f.slice/crio-4985003f74307967b7e6b56e095bbd1025e75a6213aa3da9735d87ec8ebd7d99 WatchSource:0}: Error finding container 4985003f74307967b7e6b56e095bbd1025e75a6213aa3da9735d87ec8ebd7d99: Status 404 returned error can't find the container with id 4985003f74307967b7e6b56e095bbd1025e75a6213aa3da9735d87ec8ebd7d99
Nov 03 21:57:24.441354 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:24.441225    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b2a8f46-bfh7k"]
Nov 03 21:57:24.716270 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:24.716223    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2a8f46-bfh7k" event={"ID":"018a254b-8a68-402c-a0d2-ce1fbf23796f","Type":"ContainerStarted","Data":"4985003f74307967b7e6b56e095bbd1025e75a6213aa3da9735d87ec8ebd7d99"}
Nov 03 21:57:25.725487 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:25.725454    5053 generic.go:356] "Generic (PLEG): container finished" podID="018a254b-8a68-402c-a0d2-ce1fbf23796f" containerID="d9bf3f37c02f73226a5d3b8aee910e07feb8d84686929e8b7eb8dc1db4ecc403" exitCode=0
Nov 03 21:57:25.725889 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:25.725542    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2a8f46-bfh7k" event={"ID":"018a254b-8a68-402c-a0d2-ce1fbf23796f","Type":"ContainerDied","Data":"d9bf3f37c02f73226a5d3b8aee910e07feb8d84686929e8b7eb8dc1db4ecc403"}
Nov 03 21:57:26.880547 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:26.880516    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2a8f46-bfh7k"
Nov 03 21:57:27.027022 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:27.026935    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/018a254b-8a68-402c-a0d2-ce1fbf23796f-kube-api-access-m8n4m\" (UniqueName: \"kubernetes.io/projected/018a254b-8a68-402c-a0d2-ce1fbf23796f-kube-api-access-m8n4m\") pod \"018a254b-8a68-402c-a0d2-ce1fbf23796f\" (UID: \"018a254b-8a68-402c-a0d2-ce1fbf23796f\") "
Nov 03 21:57:27.027182 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:27.027110    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/018a254b-8a68-402c-a0d2-ce1fbf23796f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/018a254b-8a68-402c-a0d2-ce1fbf23796f-maas-api-xks-ca\") pod \"018a254b-8a68-402c-a0d2-ce1fbf23796f\" (UID: \"018a254b-8a68-402c-a0d2-ce1fbf23796f\") "
Nov 03 21:57:27.027182 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:27.027164    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/018a254b-8a68-402c-a0d2-ce1fbf23796f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/018a254b-8a68-402c-a0d2-ce1fbf23796f-maas-api-openshift-ca\") pod \"018a254b-8a68-402c-a0d2-ce1fbf23796f\" (UID: \"018a254b-8a68-402c-a0d2-ce1fbf23796f\") "
Nov 03 21:57:27.027643 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:27.027613    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/018a254b-8a68-402c-a0d2-ce1fbf23796f-maas-api-openshift-ca" pod "018a254b-8a68-402c-a0d2-ce1fbf23796f" (UID: "018a254b-8a68-402c-a0d2-ce1fbf23796f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:57:27.029344 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:27.029314    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/018a254b-8a68-402c-a0d2-ce1fbf23796f-kube-api-access-m8n4m" pod "018a254b-8a68-402c-a0d2-ce1fbf23796f" (UID: "018a254b-8a68-402c-a0d2-ce1fbf23796f"). InnerVolumeSpecName "kube-api-access-m8n4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:57:27.029737 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:27.029718    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/018a254b-8a68-402c-a0d2-ce1fbf23796f-maas-api-xks-ca" pod "018a254b-8a68-402c-a0d2-ce1fbf23796f" (UID: "018a254b-8a68-402c-a0d2-ce1fbf23796f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:57:27.129263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:27.129207    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/018a254b-8a68-402c-a0d2-ce1fbf23796f-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:27.129263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:27.129249    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/018a254b-8a68-402c-a0d2-ce1fbf23796f-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:27.129263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:27.129260    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8n4m\" (UniqueName: \"kubernetes.io/projected/018a254b-8a68-402c-a0d2-ce1fbf23796f-kube-api-access-m8n4m\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:27.741412 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:27.741375    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2a8f46-bfh7k" event={"ID":"018a254b-8a68-402c-a0d2-ce1fbf23796f","Type":"ContainerDied","Data":"4985003f74307967b7e6b56e095bbd1025e75a6213aa3da9735d87ec8ebd7d99"}
Nov 03 21:57:27.741627 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:27.741435    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4985003f74307967b7e6b56e095bbd1025e75a6213aa3da9735d87ec8ebd7d99"
Nov 03 21:57:27.741627 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:27.741390    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2a8f46-bfh7k"
Nov 03 21:57:29.045546 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:29.045512    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"]
Nov 03 21:57:29.046018 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:29.045792    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4" podUID="d0510a33-96d0-4ad8-a5f3-3dba5a009fc7" containerName="istio-proxy" containerID="cri-o://6df2db91c842d2ff7b902e402008a66b8fe0829f36fcf10f87172b3317284d3a" gracePeriod=30
Nov 03 21:57:29.991029 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:29.990990    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8"]
Nov 03 21:57:29.991629 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:29.991615    5053 cpu_manager.go:414] "containerMap: removing container" podUID="018a254b-8a68-402c-a0d2-ce1fbf23796f" containerName="revoke-keys"
Nov 03 21:57:29.991686 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:29.991631    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="018a254b-8a68-402c-a0d2-ce1fbf23796f" containerName="revoke-keys"
Nov 03 21:57:29.991769 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:29.991760    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="018a254b-8a68-402c-a0d2-ce1fbf23796f" containerName="revoke-keys"
Nov 03 21:57:29.994711 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:29.994693    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8"
Nov 03 21:57:30.051125 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:30.051090    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8"]
Nov 03 21:57:30.095957 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:30.095926    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7add5909-a08d-4dba-b2f4-37c44cee1d9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8\" (UID: \"7add5909-a08d-4dba-b2f4-37c44cee1d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8"
Nov 03 21:57:30.096155 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:30.095962    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mbtg\" (UniqueName: \"kubernetes.io/projected/7add5909-a08d-4dba-b2f4-37c44cee1d9d-kube-api-access-9mbtg\") pod \"maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8\" (UID: \"7add5909-a08d-4dba-b2f4-37c44cee1d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8"
Nov 03 21:57:30.096210 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:30.096159    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7add5909-a08d-4dba-b2f4-37c44cee1d9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8\" (UID: \"7add5909-a08d-4dba-b2f4-37c44cee1d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8"
Nov 03 21:57:30.096478 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:30.096457    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:57:30.096573 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:30.096527    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:57:30.197387 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:30.197347    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7add5909-a08d-4dba-b2f4-37c44cee1d9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8\" (UID: \"7add5909-a08d-4dba-b2f4-37c44cee1d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8"
Nov 03 21:57:30.197558 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:30.197484    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7add5909-a08d-4dba-b2f4-37c44cee1d9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8\" (UID: \"7add5909-a08d-4dba-b2f4-37c44cee1d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8"
Nov 03 21:57:30.197558 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:30.197506    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mbtg\" (UniqueName: \"kubernetes.io/projected/7add5909-a08d-4dba-b2f4-37c44cee1d9d-kube-api-access-9mbtg\") pod \"maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8\" (UID: \"7add5909-a08d-4dba-b2f4-37c44cee1d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8"
Nov 03 21:57:30.198109 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:30.198084    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7add5909-a08d-4dba-b2f4-37c44cee1d9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8\" (UID: \"7add5909-a08d-4dba-b2f4-37c44cee1d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8"
Nov 03 21:57:30.200379 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:30.200354    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7add5909-a08d-4dba-b2f4-37c44cee1d9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8\" (UID: \"7add5909-a08d-4dba-b2f4-37c44cee1d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8"
Nov 03 21:57:30.254174 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:30.254105    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mbtg\" (UniqueName: \"kubernetes.io/projected/7add5909-a08d-4dba-b2f4-37c44cee1d9d-kube-api-access-9mbtg\") pod \"maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8\" (UID: \"7add5909-a08d-4dba-b2f4-37c44cee1d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8"
Nov 03 21:57:30.307863 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:30.307829    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8"
Nov 03 21:57:30.703496 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:30.703461    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8"]
Nov 03 21:57:30.704600 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:57:30.704570    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7add5909_a08d_4dba_b2f4_37c44cee1d9d.slice/crio-0628518a73bfb9febcdf556c5f577d45623c4762031888a56a0bcac018a17821 WatchSource:0}: Error finding container 0628518a73bfb9febcdf556c5f577d45623c4762031888a56a0bcac018a17821: Status 404 returned error can't find the container with id 0628518a73bfb9febcdf556c5f577d45623c4762031888a56a0bcac018a17821
Nov 03 21:57:30.761784 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:30.761755    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8" event={"ID":"7add5909-a08d-4dba-b2f4-37c44cee1d9d","Type":"ContainerStarted","Data":"0628518a73bfb9febcdf556c5f577d45623c4762031888a56a0bcac018a17821"}
Nov 03 21:57:31.770985 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:31.770946    5053 generic.go:356] "Generic (PLEG): container finished" podID="7add5909-a08d-4dba-b2f4-37c44cee1d9d" containerID="7997c4f2b762321498668018d0fc3b6ae73ea1cfc5cb889c8dcd787c65c745c3" exitCode=0
Nov 03 21:57:31.771508 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:31.770999    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8" event={"ID":"7add5909-a08d-4dba-b2f4-37c44cee1d9d","Type":"ContainerDied","Data":"7997c4f2b762321498668018d0fc3b6ae73ea1cfc5cb889c8dcd787c65c745c3"}
Nov 03 21:57:32.937736 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:32.937707    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8"
Nov 03 21:57:33.030194 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:33.030105    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7add5909-a08d-4dba-b2f4-37c44cee1d9d-kube-api-access-9mbtg\" (UniqueName: \"kubernetes.io/projected/7add5909-a08d-4dba-b2f4-37c44cee1d9d-kube-api-access-9mbtg\") pod \"7add5909-a08d-4dba-b2f4-37c44cee1d9d\" (UID: \"7add5909-a08d-4dba-b2f4-37c44cee1d9d\") "
Nov 03 21:57:33.030371 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:33.030328    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7add5909-a08d-4dba-b2f4-37c44cee1d9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7add5909-a08d-4dba-b2f4-37c44cee1d9d-maas-api-openshift-ca\") pod \"7add5909-a08d-4dba-b2f4-37c44cee1d9d\" (UID: \"7add5909-a08d-4dba-b2f4-37c44cee1d9d\") "
Nov 03 21:57:33.030487 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:33.030462    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7add5909-a08d-4dba-b2f4-37c44cee1d9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7add5909-a08d-4dba-b2f4-37c44cee1d9d-maas-api-xks-ca\") pod \"7add5909-a08d-4dba-b2f4-37c44cee1d9d\" (UID: \"7add5909-a08d-4dba-b2f4-37c44cee1d9d\") "
Nov 03 21:57:33.030718 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:33.030693    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7add5909-a08d-4dba-b2f4-37c44cee1d9d-maas-api-openshift-ca" pod "7add5909-a08d-4dba-b2f4-37c44cee1d9d" (UID: "7add5909-a08d-4dba-b2f4-37c44cee1d9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:57:33.030966 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:33.030941    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7add5909-a08d-4dba-b2f4-37c44cee1d9d-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:33.032498 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:33.032472    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7add5909-a08d-4dba-b2f4-37c44cee1d9d-kube-api-access-9mbtg" pod "7add5909-a08d-4dba-b2f4-37c44cee1d9d" (UID: "7add5909-a08d-4dba-b2f4-37c44cee1d9d"). InnerVolumeSpecName "kube-api-access-9mbtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:57:33.032632 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:33.032615    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7add5909-a08d-4dba-b2f4-37c44cee1d9d-maas-api-xks-ca" pod "7add5909-a08d-4dba-b2f4-37c44cee1d9d" (UID: "7add5909-a08d-4dba-b2f4-37c44cee1d9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:57:33.132753 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:33.132720    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7add5909-a08d-4dba-b2f4-37c44cee1d9d-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:33.132753 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:33.132748    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mbtg\" (UniqueName: \"kubernetes.io/projected/7add5909-a08d-4dba-b2f4-37c44cee1d9d-kube-api-access-9mbtg\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:33.785307 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:33.785265    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8"
Nov 03 21:57:33.785454 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:33.785262    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8" event={"ID":"7add5909-a08d-4dba-b2f4-37c44cee1d9d","Type":"ContainerDied","Data":"0628518a73bfb9febcdf556c5f577d45623c4762031888a56a0bcac018a17821"}
Nov 03 21:57:33.785454 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:33.785398    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0628518a73bfb9febcdf556c5f577d45623c4762031888a56a0bcac018a17821"
Nov 03 21:57:34.327425 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.327398    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:57:34.349576 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.349542    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-workload-socket\") pod \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") "
Nov 03 21:57:34.349717 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.349654    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-kube-api-access-swxcc\" (UniqueName: \"kubernetes.io/projected/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-kube-api-access-swxcc\") pod \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") "
Nov 03 21:57:34.349777 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.349762    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istiod-ca-cert\") pod \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") "
Nov 03 21:57:34.349813 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.349787    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-credential-socket\") pod \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") "
Nov 03 21:57:34.349813 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.349803    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-data\") pod \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") "
Nov 03 21:57:34.349919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.349851    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-envoy\") pod \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") "
Nov 03 21:57:34.349919 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.349851    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-workload-socket" pod "d0510a33-96d0-4ad8-a5f3-3dba5a009fc7" (UID: "d0510a33-96d0-4ad8-a5f3-3dba5a009fc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:57:34.350017 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.349921    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-token\" (UniqueName: \"kubernetes.io/projected/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-token\") pod \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") "
Nov 03 21:57:34.350017 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.349968    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-workload-certs\") pod \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") "
Nov 03 21:57:34.350116 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.350020    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-podinfo\") pod \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\" (UID: \"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7\") "
Nov 03 21:57:34.350116 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.350078    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-credential-socket" pod "d0510a33-96d0-4ad8-a5f3-3dba5a009fc7" (UID: "d0510a33-96d0-4ad8-a5f3-3dba5a009fc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:57:34.350116 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.350088    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-data" pod "d0510a33-96d0-4ad8-a5f3-3dba5a009fc7" (UID: "d0510a33-96d0-4ad8-a5f3-3dba5a009fc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:57:34.350295 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.350171    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istiod-ca-cert" pod "d0510a33-96d0-4ad8-a5f3-3dba5a009fc7" (UID: "d0510a33-96d0-4ad8-a5f3-3dba5a009fc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:57:34.350357 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.350340    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-workload-certs" pod "d0510a33-96d0-4ad8-a5f3-3dba5a009fc7" (UID: "d0510a33-96d0-4ad8-a5f3-3dba5a009fc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:57:34.350586 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.350563    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:34.350586 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.350589    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:34.350809 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.350603    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:34.350809 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.350616    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:34.350809 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.350629    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:34.352458 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.352428    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-podinfo" pod "d0510a33-96d0-4ad8-a5f3-3dba5a009fc7" (UID: "d0510a33-96d0-4ad8-a5f3-3dba5a009fc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:57:34.352618 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.352508    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-token" pod "d0510a33-96d0-4ad8-a5f3-3dba5a009fc7" (UID: "d0510a33-96d0-4ad8-a5f3-3dba5a009fc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:57:34.352735 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.352715    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-kube-api-access-swxcc" pod "d0510a33-96d0-4ad8-a5f3-3dba5a009fc7" (UID: "d0510a33-96d0-4ad8-a5f3-3dba5a009fc7"). InnerVolumeSpecName "kube-api-access-swxcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:57:34.352796 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.352760    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-envoy" pod "d0510a33-96d0-4ad8-a5f3-3dba5a009fc7" (UID: "d0510a33-96d0-4ad8-a5f3-3dba5a009fc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:57:34.452195 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.452161    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:34.452195 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.452192    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:34.452195 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.452202    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:34.452543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.452213    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swxcc\" (UniqueName: \"kubernetes.io/projected/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7-kube-api-access-swxcc\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:34.793882 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.793848    5053 generic.go:356] "Generic (PLEG): container finished" podID="d0510a33-96d0-4ad8-a5f3-3dba5a009fc7" containerID="6df2db91c842d2ff7b902e402008a66b8fe0829f36fcf10f87172b3317284d3a" exitCode=0
Nov 03 21:57:34.794044 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.793894    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"
Nov 03 21:57:34.794044 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.793922    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4" event={"ID":"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7","Type":"ContainerDied","Data":"6df2db91c842d2ff7b902e402008a66b8fe0829f36fcf10f87172b3317284d3a"}
Nov 03 21:57:34.794044 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.793979    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4" event={"ID":"d0510a33-96d0-4ad8-a5f3-3dba5a009fc7","Type":"ContainerDied","Data":"d3df071e6877371851e4e24d912e594c8a0959d4c6255e4d7f21f0483b1df7d3"}
Nov 03 21:57:34.794044 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.794003    5053 scope.go:122] "RemoveContainer" containerID="6df2db91c842d2ff7b902e402008a66b8fe0829f36fcf10f87172b3317284d3a"
Nov 03 21:57:34.811366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.811223    5053 scope.go:122] "RemoveContainer" containerID="9c350f8575b4e372536a24c13a1a58240f6d99afbfc8149d4f4ca0cabcff5078"
Nov 03 21:57:34.822450 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.822422    5053 scope.go:122] "RemoveContainer" containerID="6df2db91c842d2ff7b902e402008a66b8fe0829f36fcf10f87172b3317284d3a"
Nov 03 21:57:34.822766 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:57:34.822744    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df2db91c842d2ff7b902e402008a66b8fe0829f36fcf10f87172b3317284d3a\": container with ID starting with 6df2db91c842d2ff7b902e402008a66b8fe0829f36fcf10f87172b3317284d3a not found: ID does not exist" containerID="6df2db91c842d2ff7b902e402008a66b8fe0829f36fcf10f87172b3317284d3a"
Nov 03 21:57:34.822827 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.822778    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df2db91c842d2ff7b902e402008a66b8fe0829f36fcf10f87172b3317284d3a"} err="failed to get container status \"6df2db91c842d2ff7b902e402008a66b8fe0829f36fcf10f87172b3317284d3a\": rpc error: code = NotFound desc = could not find container \"6df2db91c842d2ff7b902e402008a66b8fe0829f36fcf10f87172b3317284d3a\": container with ID starting with 6df2db91c842d2ff7b902e402008a66b8fe0829f36fcf10f87172b3317284d3a not found: ID does not exist"
Nov 03 21:57:34.822827 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.822801    5053 scope.go:122] "RemoveContainer" containerID="9c350f8575b4e372536a24c13a1a58240f6d99afbfc8149d4f4ca0cabcff5078"
Nov 03 21:57:34.823050 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:57:34.823031    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c350f8575b4e372536a24c13a1a58240f6d99afbfc8149d4f4ca0cabcff5078\": container with ID starting with 9c350f8575b4e372536a24c13a1a58240f6d99afbfc8149d4f4ca0cabcff5078 not found: ID does not exist" containerID="9c350f8575b4e372536a24c13a1a58240f6d99afbfc8149d4f4ca0cabcff5078"
Nov 03 21:57:34.823101 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.823056    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c350f8575b4e372536a24c13a1a58240f6d99afbfc8149d4f4ca0cabcff5078"} err="failed to get container status \"9c350f8575b4e372536a24c13a1a58240f6d99afbfc8149d4f4ca0cabcff5078\": rpc error: code = NotFound desc = could not find container \"9c350f8575b4e372536a24c13a1a58240f6d99afbfc8149d4f4ca0cabcff5078\": container with ID starting with 9c350f8575b4e372536a24c13a1a58240f6d99afbfc8149d4f4ca0cabcff5078 not found: ID does not exist"
Nov 03 21:57:34.959018 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:34.958986    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"]
Nov 03 21:57:35.010455 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:35.010420    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-ca758d-openshift-default-75c746977c-6ckd4"]
Nov 03 21:57:35.667789 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:57:35.667757    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0510a33-96d0-4ad8-a5f3-3dba5a009fc7" path="/var/lib/kubelet/pods/d0510a33-96d0-4ad8-a5f3-3dba5a009fc7/volumes"
Nov 03 21:57:40.930265 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:57:40.930196    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:58:03.012150 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.012114    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"]
Nov 03 21:58:03.013040 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.013020    5053 cpu_manager.go:414] "containerMap: removing container" podUID="d0510a33-96d0-4ad8-a5f3-3dba5a009fc7" containerName="istio-proxy"
Nov 03 21:58:03.013095 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.013042    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0510a33-96d0-4ad8-a5f3-3dba5a009fc7" containerName="istio-proxy"
Nov 03 21:58:03.013095 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.013064    5053 cpu_manager.go:414] "containerMap: removing container" podUID="d0510a33-96d0-4ad8-a5f3-3dba5a009fc7" containerName="istio-proxy"
Nov 03 21:58:03.013095 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.013069    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0510a33-96d0-4ad8-a5f3-3dba5a009fc7" containerName="istio-proxy"
Nov 03 21:58:03.013095 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.013081    5053 cpu_manager.go:414] "containerMap: removing container" podUID="7add5909-a08d-4dba-b2f4-37c44cee1d9d" containerName="revoke-keys"
Nov 03 21:58:03.013095 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.013086    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7add5909-a08d-4dba-b2f4-37c44cee1d9d" containerName="revoke-keys"
Nov 03 21:58:03.013286 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.013198    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0510a33-96d0-4ad8-a5f3-3dba5a009fc7" containerName="istio-proxy"
Nov 03 21:58:03.013286 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.013209    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0510a33-96d0-4ad8-a5f3-3dba5a009fc7" containerName="istio-proxy"
Nov 03 21:58:03.013286 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.013215    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="7add5909-a08d-4dba-b2f4-37c44cee1d9d" containerName="revoke-keys"
Nov 03 21:58:03.018225 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.018204    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.037784 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.037741    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-istio-envoy\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.037970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.037830    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-workload-certs\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.037970 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.037955    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720e4837-f945-4df0-a637-c2dc41608b7e-istiod-ca-cert\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.038095 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.038025    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-istio-data\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.038152 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.038102    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720e4837-f945-4df0-a637-c2dc41608b7e-istio-token\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.038152 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.038131    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720e4837-f945-4df0-a637-c2dc41608b7e-istio-podinfo\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.038337 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.038315    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-credential-socket\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.038404 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.038368    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-workload-socket\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.038460 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.038421    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv9nw\" (UniqueName: \"kubernetes.io/projected/720e4837-f945-4df0-a637-c2dc41608b7e-kube-api-access-tv9nw\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.070637 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.070600    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-4644ae-openshift-default-dockercfg-vp6pk\""
Nov 03 21:58:03.186465 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.186432    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720e4837-f945-4df0-a637-c2dc41608b7e-istiod-ca-cert\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.207020 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.195227    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720e4837-f945-4df0-a637-c2dc41608b7e-istiod-ca-cert\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.207306 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.207078    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-istio-data\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.207306 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.207132    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720e4837-f945-4df0-a637-c2dc41608b7e-istio-token\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.207306 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.207150    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720e4837-f945-4df0-a637-c2dc41608b7e-istio-podinfo\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.207306 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.207284    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-credential-socket\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.207541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.207326    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-workload-socket\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.207541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.207365    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv9nw\" (UniqueName: \"kubernetes.io/projected/720e4837-f945-4df0-a637-c2dc41608b7e-kube-api-access-tv9nw\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.207541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.207420    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-istio-envoy\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.207541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.207467    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-workload-certs\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.207795 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.207778    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-workload-certs\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.207950 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.207935    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-istio-data\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.208669 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.208648    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-workload-socket\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.208824 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.208806    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-credential-socket\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.213700 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.213656    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"]
Nov 03 21:58:03.220585 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.220556    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720e4837-f945-4df0-a637-c2dc41608b7e-istio-podinfo\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.221138 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.221115    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-istio-envoy\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.270436 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.270360    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720e4837-f945-4df0-a637-c2dc41608b7e-istio-token\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.281839 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.281807    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv9nw\" (UniqueName: \"kubernetes.io/projected/720e4837-f945-4df0-a637-c2dc41608b7e-kube-api-access-tv9nw\") pod \"e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") " pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.332750 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.332708    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:03.766440 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.766413    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"]
Nov 03 21:58:03.767528 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:58:03.767500    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720e4837_f945_4df0_a637_c2dc41608b7e.slice/crio-c4f3351c099e67b848450116eaa62ac911a917fd6ca9b887b538671881b53a21 WatchSource:0}: Error finding container c4f3351c099e67b848450116eaa62ac911a917fd6ca9b887b538671881b53a21: Status 404 returned error can't find the container with id c4f3351c099e67b848450116eaa62ac911a917fd6ca9b887b538671881b53a21
Nov 03 21:58:03.770062 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.770031    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:58:03.770133 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.770098    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:58:03.770133 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:03.770126    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:58:04.015307 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:04.015204    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk" event={"ID":"720e4837-f945-4df0-a637-c2dc41608b7e","Type":"ContainerStarted","Data":"1f4e9cb903c04d892ac73fdfb0537b6103928248909ed4aa97b5c97b7367f471"}
Nov 03 21:58:04.015307 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:04.015286    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk" event={"ID":"720e4837-f945-4df0-a637-c2dc41608b7e","Type":"ContainerStarted","Data":"c4f3351c099e67b848450116eaa62ac911a917fd6ca9b887b538671881b53a21"}
Nov 03 21:58:04.132665 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:04.132606    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk" podStartSLOduration=2.132586928 podStartE2EDuration="2.132586928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:58:04.131400728 +0000 UTC m=+2598.625322614" watchObservedRunningTime="2026-11-03 21:58:04.132586928 +0000 UTC m=+2598.626508802"
Nov 03 21:58:04.332977 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:04.332881    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:04.338853 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:04.338825    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:04.354628 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:04.354599    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8"]
Nov 03 21:58:04.358968 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:04.358948    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8"
Nov 03 21:58:04.411979 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:04.411940    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:58:04.412427 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:04.412380    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:58:04.415263 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:04.415217    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8"]
Nov 03 21:58:04.426027 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:04.425633    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4569cae-30e4-48ff-a5be-25700240eaeb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8\" (UID: \"d4569cae-30e4-48ff-a5be-25700240eaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8"
Nov 03 21:58:04.426027 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:04.425681    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gss9n\" (UniqueName: \"kubernetes.io/projected/d4569cae-30e4-48ff-a5be-25700240eaeb-kube-api-access-gss9n\") pod \"maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8\" (UID: \"d4569cae-30e4-48ff-a5be-25700240eaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8"
Nov 03 21:58:04.426027 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:04.425731    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4569cae-30e4-48ff-a5be-25700240eaeb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8\" (UID: \"d4569cae-30e4-48ff-a5be-25700240eaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8"
Nov 03 21:58:04.526935 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:04.526895    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4569cae-30e4-48ff-a5be-25700240eaeb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8\" (UID: \"d4569cae-30e4-48ff-a5be-25700240eaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8"
Nov 03 21:58:04.526935 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:04.526933    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gss9n\" (UniqueName: \"kubernetes.io/projected/d4569cae-30e4-48ff-a5be-25700240eaeb-kube-api-access-gss9n\") pod \"maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8\" (UID: \"d4569cae-30e4-48ff-a5be-25700240eaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8"
Nov 03 21:58:04.527219 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:04.526960    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4569cae-30e4-48ff-a5be-25700240eaeb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8\" (UID: \"d4569cae-30e4-48ff-a5be-25700240eaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8"
Nov 03 21:58:04.527677 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:04.527647    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4569cae-30e4-48ff-a5be-25700240eaeb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8\" (UID: \"d4569cae-30e4-48ff-a5be-25700240eaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8"
Nov 03 21:58:04.529600 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:04.529574    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4569cae-30e4-48ff-a5be-25700240eaeb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8\" (UID: \"d4569cae-30e4-48ff-a5be-25700240eaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8"
Nov 03 21:58:04.583389 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:04.583316    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gss9n\" (UniqueName: \"kubernetes.io/projected/d4569cae-30e4-48ff-a5be-25700240eaeb-kube-api-access-gss9n\") pod \"maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8\" (UID: \"d4569cae-30e4-48ff-a5be-25700240eaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8"
Nov 03 21:58:04.675544 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:04.675512    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8"
Nov 03 21:58:05.022088 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:05.022053    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:05.023504 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:05.023485    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:58:05.057053 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:05.057028    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8"]
Nov 03 21:58:05.057440 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:58:05.057417    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4569cae_30e4_48ff_a5be_25700240eaeb.slice/crio-20b04fb015a40f7199fc92a3d9a8af60f824f21935301734ccaad0cc895f109c WatchSource:0}: Error finding container 20b04fb015a40f7199fc92a3d9a8af60f824f21935301734ccaad0cc895f109c: Status 404 returned error can't find the container with id 20b04fb015a40f7199fc92a3d9a8af60f824f21935301734ccaad0cc895f109c
Nov 03 21:58:06.031881 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:06.031845    5053 generic.go:356] "Generic (PLEG): container finished" podID="d4569cae-30e4-48ff-a5be-25700240eaeb" containerID="ce97895737408d7f79eec730b139bfaa480270b1edabd40bf5112810ba4e490a" exitCode=0
Nov 03 21:58:06.032298 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:06.031894    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8" event={"ID":"d4569cae-30e4-48ff-a5be-25700240eaeb","Type":"ContainerDied","Data":"ce97895737408d7f79eec730b139bfaa480270b1edabd40bf5112810ba4e490a"}
Nov 03 21:58:06.032298 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:06.031966    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8" event={"ID":"d4569cae-30e4-48ff-a5be-25700240eaeb","Type":"ContainerStarted","Data":"20b04fb015a40f7199fc92a3d9a8af60f824f21935301734ccaad0cc895f109c"}
Nov 03 21:58:07.209256 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:07.209213    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8"
Nov 03 21:58:07.262885 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:07.262845    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4569cae-30e4-48ff-a5be-25700240eaeb-kube-api-access-gss9n\" (UniqueName: \"kubernetes.io/projected/d4569cae-30e4-48ff-a5be-25700240eaeb-kube-api-access-gss9n\") pod \"d4569cae-30e4-48ff-a5be-25700240eaeb\" (UID: \"d4569cae-30e4-48ff-a5be-25700240eaeb\") "
Nov 03 21:58:07.263097 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:07.262937    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4569cae-30e4-48ff-a5be-25700240eaeb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4569cae-30e4-48ff-a5be-25700240eaeb-maas-api-openshift-ca\") pod \"d4569cae-30e4-48ff-a5be-25700240eaeb\" (UID: \"d4569cae-30e4-48ff-a5be-25700240eaeb\") "
Nov 03 21:58:07.263097 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:07.262992    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4569cae-30e4-48ff-a5be-25700240eaeb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4569cae-30e4-48ff-a5be-25700240eaeb-maas-api-xks-ca\") pod \"d4569cae-30e4-48ff-a5be-25700240eaeb\" (UID: \"d4569cae-30e4-48ff-a5be-25700240eaeb\") "
Nov 03 21:58:07.263450 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:07.263426    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4569cae-30e4-48ff-a5be-25700240eaeb-maas-api-openshift-ca" pod "d4569cae-30e4-48ff-a5be-25700240eaeb" (UID: "d4569cae-30e4-48ff-a5be-25700240eaeb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:58:07.265155 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:07.265131    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4569cae-30e4-48ff-a5be-25700240eaeb-maas-api-xks-ca" pod "d4569cae-30e4-48ff-a5be-25700240eaeb" (UID: "d4569cae-30e4-48ff-a5be-25700240eaeb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:58:07.265285 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:07.265162    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4569cae-30e4-48ff-a5be-25700240eaeb-kube-api-access-gss9n" pod "d4569cae-30e4-48ff-a5be-25700240eaeb" (UID: "d4569cae-30e4-48ff-a5be-25700240eaeb"). InnerVolumeSpecName "kube-api-access-gss9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:58:07.364525 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:07.364442    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gss9n\" (UniqueName: \"kubernetes.io/projected/d4569cae-30e4-48ff-a5be-25700240eaeb-kube-api-access-gss9n\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:07.364525 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:07.364479    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4569cae-30e4-48ff-a5be-25700240eaeb-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:07.364525 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:07.364497    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4569cae-30e4-48ff-a5be-25700240eaeb-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:08.048967 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:08.048936    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8"
Nov 03 21:58:08.049156 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:08.048968    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8" event={"ID":"d4569cae-30e4-48ff-a5be-25700240eaeb","Type":"ContainerDied","Data":"20b04fb015a40f7199fc92a3d9a8af60f824f21935301734ccaad0cc895f109c"}
Nov 03 21:58:08.049156 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:08.049033    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20b04fb015a40f7199fc92a3d9a8af60f824f21935301734ccaad0cc895f109c"
Nov 03 21:58:08.316042 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:08.315935    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"]
Nov 03 21:58:08.316626 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:08.316607    5053 cpu_manager.go:414] "containerMap: removing container" podUID="d4569cae-30e4-48ff-a5be-25700240eaeb" containerName="revoke-keys"
Nov 03 21:58:08.316626 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:08.316627    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4569cae-30e4-48ff-a5be-25700240eaeb" containerName="revoke-keys"
Nov 03 21:58:08.316734 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:08.316725    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4569cae-30e4-48ff-a5be-25700240eaeb" containerName="revoke-keys"
Nov 03 21:58:08.320204 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:08.320183    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"
Nov 03 21:58:08.372142 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:08.372115    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-93c833-serving-cert\""
Nov 03 21:58:08.378760 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:08.378732    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx82l\" (UniqueName: \"kubernetes.io/projected/34ee5ddf-3c64-4a43-962f-834963d7452a-kube-api-access-bx82l\") pod \"maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l\" (UID: \"34ee5ddf-3c64-4a43-962f-834963d7452a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"
Nov 03 21:58:08.378842 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:08.378776    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34ee5ddf-3c64-4a43-962f-834963d7452a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l\" (UID: \"34ee5ddf-3c64-4a43-962f-834963d7452a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"
Nov 03 21:58:08.378880 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:08.378849    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ee5ddf-3c64-4a43-962f-834963d7452a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l\" (UID: \"34ee5ddf-3c64-4a43-962f-834963d7452a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"
Nov 03 21:58:08.378994 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:08.378977    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34ee5ddf-3c64-4a43-962f-834963d7452a-metrics-certs\") pod \"maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l\" (UID: \"34ee5ddf-3c64-4a43-962f-834963d7452a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"
Nov 03 21:58:08.418978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:08.418947    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"]
Nov 03 21:58:08.480740 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:08.480688    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx82l\" (UniqueName: \"kubernetes.io/projected/34ee5ddf-3c64-4a43-962f-834963d7452a-kube-api-access-bx82l\") pod \"maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l\" (UID: \"34ee5ddf-3c64-4a43-962f-834963d7452a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"
Nov 03 21:58:08.480942 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:08.480773    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34ee5ddf-3c64-4a43-962f-834963d7452a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l\" (UID: \"34ee5ddf-3c64-4a43-962f-834963d7452a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"
Nov 03 21:58:08.480942 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:08.480842    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ee5ddf-3c64-4a43-962f-834963d7452a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l\" (UID: \"34ee5ddf-3c64-4a43-962f-834963d7452a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"
Nov 03 21:58:08.481095 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:08.481000    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34ee5ddf-3c64-4a43-962f-834963d7452a-metrics-certs\") pod \"maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l\" (UID: \"34ee5ddf-3c64-4a43-962f-834963d7452a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"
Nov 03 21:58:08.481296 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:58:08.481224    5053 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-serving-cert: secret "maas-api-e2e-praxis-a-93c833-serving-cert" not found
Nov 03 21:58:08.481390 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:58:08.481343    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34ee5ddf-3c64-4a43-962f-834963d7452a-maas-api-tls podName:34ee5ddf-3c64-4a43-962f-834963d7452a nodeName:}" failed. No retries permitted until 2026-11-03 21:58:08.981323599 +0000 UTC m=+2603.475245455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/34ee5ddf-3c64-4a43-962f-834963d7452a-maas-api-tls") pod "maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l" (UID: "34ee5ddf-3c64-4a43-962f-834963d7452a") : secret "maas-api-e2e-praxis-a-93c833-serving-cert" not found
Nov 03 21:58:08.482222 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:08.482189    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ee5ddf-3c64-4a43-962f-834963d7452a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l\" (UID: \"34ee5ddf-3c64-4a43-962f-834963d7452a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"
Nov 03 21:58:08.484566 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:08.484538    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34ee5ddf-3c64-4a43-962f-834963d7452a-metrics-certs\") pod \"maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l\" (UID: \"34ee5ddf-3c64-4a43-962f-834963d7452a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"
Nov 03 21:58:08.547210 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:08.541778    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx82l\" (UniqueName: \"kubernetes.io/projected/34ee5ddf-3c64-4a43-962f-834963d7452a-kube-api-access-bx82l\") pod \"maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l\" (UID: \"34ee5ddf-3c64-4a43-962f-834963d7452a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"
Nov 03 21:58:08.990276 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:08.990211    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34ee5ddf-3c64-4a43-962f-834963d7452a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l\" (UID: \"34ee5ddf-3c64-4a43-962f-834963d7452a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"
Nov 03 21:58:08.993203 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:08.993170    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34ee5ddf-3c64-4a43-962f-834963d7452a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l\" (UID: \"34ee5ddf-3c64-4a43-962f-834963d7452a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"
Nov 03 21:58:09.239091 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:09.239039    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"
Nov 03 21:58:09.615004 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:09.614973    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"]
Nov 03 21:58:09.617369 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:58:09.617336    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ee5ddf_3c64_4a43_962f_834963d7452a.slice/crio-2b8f8021527a92f3ddbfe68a61f592ebb547bac741867465be9797184d1fc23e WatchSource:0}: Error finding container 2b8f8021527a92f3ddbfe68a61f592ebb547bac741867465be9797184d1fc23e: Status 404 returned error can't find the container with id 2b8f8021527a92f3ddbfe68a61f592ebb547bac741867465be9797184d1fc23e
Nov 03 21:58:09.619391 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:09.619370    5053 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:58:10.063533 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:10.063496    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l" event={"ID":"34ee5ddf-3c64-4a43-962f-834963d7452a","Type":"ContainerStarted","Data":"2b8f8021527a92f3ddbfe68a61f592ebb547bac741867465be9797184d1fc23e"}
Nov 03 21:58:10.930544 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:58:10.930438    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:58:11.073120 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:11.073084    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l" event={"ID":"34ee5ddf-3c64-4a43-962f-834963d7452a","Type":"ContainerStarted","Data":"9ecc5675229be308c9993b599d9487d606136ebfb5036961609b1156bcd1c7f5"}
Nov 03 21:58:11.073309 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:11.073150    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"
Nov 03 21:58:11.190280 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:11.190154    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l" podStartSLOduration=2.262188436 podStartE2EDuration="3.190140654s" totalImagesPullingTime="927.952218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:58:11.189360695 +0000 UTC m=+2605.683282571" watchObservedRunningTime="2026-11-03 21:58:11.190140654 +0000 UTC m=+2605.684062528"
Nov 03 21:58:17.086922 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:17.086888    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"
Nov 03 21:58:40.639839 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:40.639803    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"]
Nov 03 21:58:40.640358 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:40.640073    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx" podUID="909e67ce-5511-4f93-91eb-17f6a87acd84" containerName="istio-proxy" containerID="cri-o://6f2ec128abc20fdaad9e2fb701563bc0f3c7e727bf1048449095aaadec934b84" gracePeriod=30
Nov 03 21:58:40.929448 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:58:40.929358    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:58:44.155446 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:44.155407    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j"]
Nov 03 21:58:44.155906 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:44.155618    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j" podUID="0c6a1be6-612c-46f0-ad39-07a4bc90d9a1" containerName="maas-api" containerID="cri-o://67b704561ea4e0c819aceabd2f5b667c93128a4e59b65d4a09927eacdff1d0c7" gracePeriod=30
Nov 03 21:58:44.314826 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:44.314793    5053 generic.go:356] "Generic (PLEG): container finished" podID="0c6a1be6-612c-46f0-ad39-07a4bc90d9a1" containerID="67b704561ea4e0c819aceabd2f5b667c93128a4e59b65d4a09927eacdff1d0c7" exitCode=0
Nov 03 21:58:44.315046 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:44.314973    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j" event={"ID":"0c6a1be6-612c-46f0-ad39-07a4bc90d9a1","Type":"ContainerDied","Data":"67b704561ea4e0c819aceabd2f5b667c93128a4e59b65d4a09927eacdff1d0c7"}
Nov 03 21:58:44.442629 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:44.442596    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j"
Nov 03 21:58:44.505825 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:44.505799    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-openshift-service-ca\") pod \"0c6a1be6-612c-46f0-ad39-07a4bc90d9a1\" (UID: \"0c6a1be6-612c-46f0-ad39-07a4bc90d9a1\") "
Nov 03 21:58:44.505978 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:44.505835    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-kube-api-access-sggzb\" (UniqueName: \"kubernetes.io/projected/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-kube-api-access-sggzb\") pod \"0c6a1be6-612c-46f0-ad39-07a4bc90d9a1\" (UID: \"0c6a1be6-612c-46f0-ad39-07a4bc90d9a1\") "
Nov 03 21:58:44.506068 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:44.505982    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-metrics-certs\") pod \"0c6a1be6-612c-46f0-ad39-07a4bc90d9a1\" (UID: \"0c6a1be6-612c-46f0-ad39-07a4bc90d9a1\") "
Nov 03 21:58:44.506068 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:44.506018    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-maas-api-tls\") pod \"0c6a1be6-612c-46f0-ad39-07a4bc90d9a1\" (UID: \"0c6a1be6-612c-46f0-ad39-07a4bc90d9a1\") "
Nov 03 21:58:44.506205 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:44.506185    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-openshift-service-ca" pod "0c6a1be6-612c-46f0-ad39-07a4bc90d9a1" (UID: "0c6a1be6-612c-46f0-ad39-07a4bc90d9a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:58:44.506506 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:44.506487    5053 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-openshift-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:44.508221 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:44.508191    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-metrics-certs" pod "0c6a1be6-612c-46f0-ad39-07a4bc90d9a1" (UID: "0c6a1be6-612c-46f0-ad39-07a4bc90d9a1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:58:44.508345 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:44.508219    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-kube-api-access-sggzb" pod "0c6a1be6-612c-46f0-ad39-07a4bc90d9a1" (UID: "0c6a1be6-612c-46f0-ad39-07a4bc90d9a1"). InnerVolumeSpecName "kube-api-access-sggzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:58:44.508345 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:44.508247    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-maas-api-tls" pod "0c6a1be6-612c-46f0-ad39-07a4bc90d9a1" (UID: "0c6a1be6-612c-46f0-ad39-07a4bc90d9a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:58:44.607926 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:44.607891    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sggzb\" (UniqueName: \"kubernetes.io/projected/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-kube-api-access-sggzb\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:44.607926 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:44.607925    5053 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-metrics-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:44.608130 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:44.607939    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1-maas-api-tls\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:45.322166 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:45.322137    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j"
Nov 03 21:58:45.322702 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:45.322178    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j" event={"ID":"0c6a1be6-612c-46f0-ad39-07a4bc90d9a1","Type":"ContainerDied","Data":"7b3cb10d2675440728d3a201b1a05d7f112506ab3e65324edfb1fbefcb499b39"}
Nov 03 21:58:45.322702 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:45.322254    5053 scope.go:122] "RemoveContainer" containerID="67b704561ea4e0c819aceabd2f5b667c93128a4e59b65d4a09927eacdff1d0c7"
Nov 03 21:58:45.486615 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:45.486574    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j"]
Nov 03 21:58:45.489573 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:45.489543    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-292cf1-664c6c8b5c-x6g2j"]
Nov 03 21:58:45.669363 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:45.669334    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c6a1be6-612c-46f0-ad39-07a4bc90d9a1" path="/var/lib/kubelet/pods/0c6a1be6-612c-46f0-ad39-07a4bc90d9a1/volumes"
Nov 03 21:58:45.971293 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:45.971266    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:58:46.025075 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.025041    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/909e67ce-5511-4f93-91eb-17f6a87acd84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/909e67ce-5511-4f93-91eb-17f6a87acd84-istiod-ca-cert\") pod \"909e67ce-5511-4f93-91eb-17f6a87acd84\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") "
Nov 03 21:58:46.025075 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.025080    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-podinfo\") pod \"909e67ce-5511-4f93-91eb-17f6a87acd84\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") "
Nov 03 21:58:46.025381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.025116    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-workload-certs\") pod \"909e67ce-5511-4f93-91eb-17f6a87acd84\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") "
Nov 03 21:58:46.025381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.025146    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-workload-socket\") pod \"909e67ce-5511-4f93-91eb-17f6a87acd84\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") "
Nov 03 21:58:46.025381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.025322    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/909e67ce-5511-4f93-91eb-17f6a87acd84-kube-api-access-79w2h\" (UniqueName: \"kubernetes.io/projected/909e67ce-5511-4f93-91eb-17f6a87acd84-kube-api-access-79w2h\") pod \"909e67ce-5511-4f93-91eb-17f6a87acd84\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") "
Nov 03 21:58:46.025381 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.025354    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-credential-socket\") pod \"909e67ce-5511-4f93-91eb-17f6a87acd84\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") "
Nov 03 21:58:46.025598 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.025399    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-envoy\") pod \"909e67ce-5511-4f93-91eb-17f6a87acd84\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") "
Nov 03 21:58:46.025598 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.025448    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-data\") pod \"909e67ce-5511-4f93-91eb-17f6a87acd84\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") "
Nov 03 21:58:46.025598 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.025500    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-token\" (UniqueName: \"kubernetes.io/projected/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-token\") pod \"909e67ce-5511-4f93-91eb-17f6a87acd84\" (UID: \"909e67ce-5511-4f93-91eb-17f6a87acd84\") "
Nov 03 21:58:46.025598 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.025508    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-workload-certs" pod "909e67ce-5511-4f93-91eb-17f6a87acd84" (UID: "909e67ce-5511-4f93-91eb-17f6a87acd84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:58:46.025807 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.025629    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-workload-socket" pod "909e67ce-5511-4f93-91eb-17f6a87acd84" (UID: "909e67ce-5511-4f93-91eb-17f6a87acd84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:58:46.025807 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.025540    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/909e67ce-5511-4f93-91eb-17f6a87acd84-istiod-ca-cert" pod "909e67ce-5511-4f93-91eb-17f6a87acd84" (UID: "909e67ce-5511-4f93-91eb-17f6a87acd84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:58:46.025807 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.025778    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-data" pod "909e67ce-5511-4f93-91eb-17f6a87acd84" (UID: "909e67ce-5511-4f93-91eb-17f6a87acd84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:58:46.026271 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.026074    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-credential-socket" pod "909e67ce-5511-4f93-91eb-17f6a87acd84" (UID: "909e67ce-5511-4f93-91eb-17f6a87acd84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:58:46.026366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.026298    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:46.026366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.026321    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/909e67ce-5511-4f93-91eb-17f6a87acd84-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:46.026366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.026335    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:46.026366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.026347    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:46.027654 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.027630    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-envoy" pod "909e67ce-5511-4f93-91eb-17f6a87acd84" (UID: "909e67ce-5511-4f93-91eb-17f6a87acd84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:58:46.027743 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.027653    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-podinfo" pod "909e67ce-5511-4f93-91eb-17f6a87acd84" (UID: "909e67ce-5511-4f93-91eb-17f6a87acd84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:58:46.027743 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.027738    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/909e67ce-5511-4f93-91eb-17f6a87acd84-kube-api-access-79w2h" pod "909e67ce-5511-4f93-91eb-17f6a87acd84" (UID: "909e67ce-5511-4f93-91eb-17f6a87acd84"). InnerVolumeSpecName "kube-api-access-79w2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:58:46.027855 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.027804    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-token" pod "909e67ce-5511-4f93-91eb-17f6a87acd84" (UID: "909e67ce-5511-4f93-91eb-17f6a87acd84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:58:46.127533 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.127494    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79w2h\" (UniqueName: \"kubernetes.io/projected/909e67ce-5511-4f93-91eb-17f6a87acd84-kube-api-access-79w2h\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:46.127533 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.127530    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:46.127749 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.127544    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:46.127749 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.127555    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:46.127749 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.127567    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/909e67ce-5511-4f93-91eb-17f6a87acd84-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:46.366259 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.362308    5053 generic.go:356] "Generic (PLEG): container finished" podID="909e67ce-5511-4f93-91eb-17f6a87acd84" containerID="6f2ec128abc20fdaad9e2fb701563bc0f3c7e727bf1048449095aaadec934b84" exitCode=0
Nov 03 21:58:46.367079 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.367044    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx" event={"ID":"909e67ce-5511-4f93-91eb-17f6a87acd84","Type":"ContainerDied","Data":"6f2ec128abc20fdaad9e2fb701563bc0f3c7e727bf1048449095aaadec934b84"}
Nov 03 21:58:46.367329 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.367302    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx" event={"ID":"909e67ce-5511-4f93-91eb-17f6a87acd84","Type":"ContainerDied","Data":"a8dd8c8f97e3c9da0625086afd7f2d882c0433aa11c5efdc9b3de794c36a2445"}
Nov 03 21:58:46.367474 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.367342    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8"]
Nov 03 21:58:46.367474 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.367371    5053 scope.go:122] "RemoveContainer" containerID="6f2ec128abc20fdaad9e2fb701563bc0f3c7e727bf1048449095aaadec934b84"
Nov 03 21:58:46.367585 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.367541    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"
Nov 03 21:58:46.367985 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.367955    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-292cf1-kfbw8"]
Nov 03 21:58:46.393460 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.393435    5053 scope.go:122] "RemoveContainer" containerID="6f2ec128abc20fdaad9e2fb701563bc0f3c7e727bf1048449095aaadec934b84"
Nov 03 21:58:46.393784 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:58:46.393758    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f2ec128abc20fdaad9e2fb701563bc0f3c7e727bf1048449095aaadec934b84\": container with ID starting with 6f2ec128abc20fdaad9e2fb701563bc0f3c7e727bf1048449095aaadec934b84 not found: ID does not exist" containerID="6f2ec128abc20fdaad9e2fb701563bc0f3c7e727bf1048449095aaadec934b84"
Nov 03 21:58:46.393848 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.393794    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2ec128abc20fdaad9e2fb701563bc0f3c7e727bf1048449095aaadec934b84"} err="failed to get container status \"6f2ec128abc20fdaad9e2fb701563bc0f3c7e727bf1048449095aaadec934b84\": rpc error: code = NotFound desc = could not find container \"6f2ec128abc20fdaad9e2fb701563bc0f3c7e727bf1048449095aaadec934b84\": container with ID starting with 6f2ec128abc20fdaad9e2fb701563bc0f3c7e727bf1048449095aaadec934b84 not found: ID does not exist"
Nov 03 21:58:46.532354 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.532319    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"]
Nov 03 21:58:46.535287 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:46.535257    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8jgdx"]
Nov 03 21:58:47.667030 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:47.667000    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7add5909-a08d-4dba-b2f4-37c44cee1d9d" path="/var/lib/kubelet/pods/7add5909-a08d-4dba-b2f4-37c44cee1d9d/volumes"
Nov 03 21:58:47.667486 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:47.667473    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="909e67ce-5511-4f93-91eb-17f6a87acd84" path="/var/lib/kubelet/pods/909e67ce-5511-4f93-91eb-17f6a87acd84/volumes"
Nov 03 21:58:47.707657 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:47.707625    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8"]
Nov 03 21:58:47.757300 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:47.757254    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c993b8-4gbm8"]
Nov 03 21:58:49.667652 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:49.667622    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4569cae-30e4-48ff-a5be-25700240eaeb" path="/var/lib/kubelet/pods/d4569cae-30e4-48ff-a5be-25700240eaeb/volumes"
Nov 03 21:58:53.108406 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:53.108372    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"]
Nov 03 21:58:53.108852 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:53.108578    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw" podUID="7af1b21a-e734-4f08-aab1-9d8308baeff6" containerName="istio-proxy" containerID="cri-o://91cf30917c1dfdec41ade6cd918f8e919cc948c523ef643c5ce20f65c8293e16" gracePeriod=30
Nov 03 21:58:54.551168 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:54.551132    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"]
Nov 03 21:58:54.551700 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:54.551430    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s" podUID="ac387b47-5c73-4984-bda9-8b5c82af02a5" containerName="istio-proxy" containerID="cri-o://68299c7f150ba045e58575e0ba430e41fe57525755a55899426289ee4192c0d7" gracePeriod=30
Nov 03 21:58:55.380366 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:55.380331    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl"]
Nov 03 21:58:55.380963 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:55.380945    5053 cpu_manager.go:414] "containerMap: removing container" podUID="909e67ce-5511-4f93-91eb-17f6a87acd84" containerName="istio-proxy"
Nov 03 21:58:55.381007 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:55.380965    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="909e67ce-5511-4f93-91eb-17f6a87acd84" containerName="istio-proxy"
Nov 03 21:58:55.381042 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:55.381014    5053 cpu_manager.go:414] "containerMap: removing container" podUID="0c6a1be6-612c-46f0-ad39-07a4bc90d9a1" containerName="maas-api"
Nov 03 21:58:55.381042 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:55.381019    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c6a1be6-612c-46f0-ad39-07a4bc90d9a1" containerName="maas-api"
Nov 03 21:58:55.381142 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:55.381133    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c6a1be6-612c-46f0-ad39-07a4bc90d9a1" containerName="maas-api"
Nov 03 21:58:55.381178 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:55.381146    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="909e67ce-5511-4f93-91eb-17f6a87acd84" containerName="istio-proxy"
Nov 03 21:58:55.386432 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:55.386412    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl"
Nov 03 21:58:55.440579 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:55.440489    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl"]
Nov 03 21:58:55.470737 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:55.470700    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp757\" (UniqueName: \"kubernetes.io/projected/f7212b57-eddc-4cb8-8657-f3842ab5ad5d-kube-api-access-bp757\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl\" (UID: \"f7212b57-eddc-4cb8-8657-f3842ab5ad5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl"
Nov 03 21:58:55.470940 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:55.470773    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7212b57-eddc-4cb8-8657-f3842ab5ad5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl\" (UID: \"f7212b57-eddc-4cb8-8657-f3842ab5ad5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl"
Nov 03 21:58:55.471007 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:55.470973    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7212b57-eddc-4cb8-8657-f3842ab5ad5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl\" (UID: \"f7212b57-eddc-4cb8-8657-f3842ab5ad5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl"
Nov 03 21:58:55.485630 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:55.485593    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:58:55.485828 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:55.485688    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:58:55.572824 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:55.572784    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7212b57-eddc-4cb8-8657-f3842ab5ad5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl\" (UID: \"f7212b57-eddc-4cb8-8657-f3842ab5ad5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl"
Nov 03 21:58:55.573223 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:55.572892    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7212b57-eddc-4cb8-8657-f3842ab5ad5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl\" (UID: \"f7212b57-eddc-4cb8-8657-f3842ab5ad5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl"
Nov 03 21:58:55.573223 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:55.572985    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp757\" (UniqueName: \"kubernetes.io/projected/f7212b57-eddc-4cb8-8657-f3842ab5ad5d-kube-api-access-bp757\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl\" (UID: \"f7212b57-eddc-4cb8-8657-f3842ab5ad5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl"
Nov 03 21:58:55.573578 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:55.573560    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7212b57-eddc-4cb8-8657-f3842ab5ad5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl\" (UID: \"f7212b57-eddc-4cb8-8657-f3842ab5ad5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl"
Nov 03 21:58:55.575613 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:55.575592    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7212b57-eddc-4cb8-8657-f3842ab5ad5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl\" (UID: \"f7212b57-eddc-4cb8-8657-f3842ab5ad5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl"
Nov 03 21:58:55.629791 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:55.629756    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp757\" (UniqueName: \"kubernetes.io/projected/f7212b57-eddc-4cb8-8657-f3842ab5ad5d-kube-api-access-bp757\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl\" (UID: \"f7212b57-eddc-4cb8-8657-f3842ab5ad5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl"
Nov 03 21:58:55.700339 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:55.700310    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl"
Nov 03 21:58:56.076718 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:56.076677    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl"]
Nov 03 21:58:56.077765 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:58:56.077736    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7212b57_eddc_4cb8_8657_f3842ab5ad5d.slice/crio-8b335bedacbab619cd906d2e6e9b15e3ff5704b06492ed24105ee75440851bd2 WatchSource:0}: Error finding container 8b335bedacbab619cd906d2e6e9b15e3ff5704b06492ed24105ee75440851bd2: Status 404 returned error can't find the container with id 8b335bedacbab619cd906d2e6e9b15e3ff5704b06492ed24105ee75440851bd2
Nov 03 21:58:56.435351 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:56.435320    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl" event={"ID":"f7212b57-eddc-4cb8-8657-f3842ab5ad5d","Type":"ContainerStarted","Data":"82a2f80ea847d4ab602e0576ee1cbfe3a0fc051474415a8d061d841a1cbb670b"}
Nov 03 21:58:56.435513 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:56.435375    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl" event={"ID":"f7212b57-eddc-4cb8-8657-f3842ab5ad5d","Type":"ContainerStarted","Data":"8b335bedacbab619cd906d2e6e9b15e3ff5704b06492ed24105ee75440851bd2"}
Nov 03 21:58:56.551993 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:56.551940    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl" podStartSLOduration=1.551921896 podStartE2EDuration="1.551921896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:58:56.549639099 +0000 UTC m=+2651.043560991" watchObservedRunningTime="2026-11-03 21:58:56.551921896 +0000 UTC m=+2651.045843772"
Nov 03 21:58:57.444341 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:57.444304    5053 generic.go:356] "Generic (PLEG): container finished" podID="f7212b57-eddc-4cb8-8657-f3842ab5ad5d" containerID="82a2f80ea847d4ab602e0576ee1cbfe3a0fc051474415a8d061d841a1cbb670b" exitCode=0
Nov 03 21:58:57.444753 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:57.444392    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl" event={"ID":"f7212b57-eddc-4cb8-8657-f3842ab5ad5d","Type":"ContainerDied","Data":"82a2f80ea847d4ab602e0576ee1cbfe3a0fc051474415a8d061d841a1cbb670b"}
Nov 03 21:58:58.384622 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.384561    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:58:58.452473 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.452443    5053 generic.go:356] "Generic (PLEG): container finished" podID="7af1b21a-e734-4f08-aab1-9d8308baeff6" containerID="91cf30917c1dfdec41ade6cd918f8e919cc948c523ef643c5ce20f65c8293e16" exitCode=0
Nov 03 21:58:58.452876 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.452496    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"
Nov 03 21:58:58.452876 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.452534    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw" event={"ID":"7af1b21a-e734-4f08-aab1-9d8308baeff6","Type":"ContainerDied","Data":"91cf30917c1dfdec41ade6cd918f8e919cc948c523ef643c5ce20f65c8293e16"}
Nov 03 21:58:58.452876 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.452588    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw" event={"ID":"7af1b21a-e734-4f08-aab1-9d8308baeff6","Type":"ContainerDied","Data":"031a2c99d0ab7c7377d576ad143da6c44a63eac7b2941a3b87486762d14a36fe"}
Nov 03 21:58:58.452876 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.452611    5053 scope.go:122] "RemoveContainer" containerID="91cf30917c1dfdec41ade6cd918f8e919cc948c523ef643c5ce20f65c8293e16"
Nov 03 21:58:58.463297 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.463278    5053 scope.go:122] "RemoveContainer" containerID="be6c84faf0201e0f5b797f5503eb86dfabfd31931aaa21a6cc7231efcd8cb702"
Nov 03 21:58:58.475997 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.475974    5053 scope.go:122] "RemoveContainer" containerID="91cf30917c1dfdec41ade6cd918f8e919cc948c523ef643c5ce20f65c8293e16"
Nov 03 21:58:58.476409 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:58:58.476384    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91cf30917c1dfdec41ade6cd918f8e919cc948c523ef643c5ce20f65c8293e16\": container with ID starting with 91cf30917c1dfdec41ade6cd918f8e919cc948c523ef643c5ce20f65c8293e16 not found: ID does not exist" containerID="91cf30917c1dfdec41ade6cd918f8e919cc948c523ef643c5ce20f65c8293e16"
Nov 03 21:58:58.476547 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.476418    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91cf30917c1dfdec41ade6cd918f8e919cc948c523ef643c5ce20f65c8293e16"} err="failed to get container status \"91cf30917c1dfdec41ade6cd918f8e919cc948c523ef643c5ce20f65c8293e16\": rpc error: code = NotFound desc = could not find container \"91cf30917c1dfdec41ade6cd918f8e919cc948c523ef643c5ce20f65c8293e16\": container with ID starting with 91cf30917c1dfdec41ade6cd918f8e919cc948c523ef643c5ce20f65c8293e16 not found: ID does not exist"
Nov 03 21:58:58.476547 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.476438    5053 scope.go:122] "RemoveContainer" containerID="be6c84faf0201e0f5b797f5503eb86dfabfd31931aaa21a6cc7231efcd8cb702"
Nov 03 21:58:58.476699 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:58:58.476680    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be6c84faf0201e0f5b797f5503eb86dfabfd31931aaa21a6cc7231efcd8cb702\": container with ID starting with be6c84faf0201e0f5b797f5503eb86dfabfd31931aaa21a6cc7231efcd8cb702 not found: ID does not exist" containerID="be6c84faf0201e0f5b797f5503eb86dfabfd31931aaa21a6cc7231efcd8cb702"
Nov 03 21:58:58.476766 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.476700    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6c84faf0201e0f5b797f5503eb86dfabfd31931aaa21a6cc7231efcd8cb702"} err="failed to get container status \"be6c84faf0201e0f5b797f5503eb86dfabfd31931aaa21a6cc7231efcd8cb702\": rpc error: code = NotFound desc = could not find container \"be6c84faf0201e0f5b797f5503eb86dfabfd31931aaa21a6cc7231efcd8cb702\": container with ID starting with be6c84faf0201e0f5b797f5503eb86dfabfd31931aaa21a6cc7231efcd8cb702 not found: ID does not exist"
Nov 03 21:58:58.508216 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.508179    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-credential-socket\") pod \"7af1b21a-e734-4f08-aab1-9d8308baeff6\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") "
Nov 03 21:58:58.508216 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.508214    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-envoy\") pod \"7af1b21a-e734-4f08-aab1-9d8308baeff6\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") "
Nov 03 21:58:58.508503 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.508256    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af1b21a-e734-4f08-aab1-9d8308baeff6-kube-api-access-fz6l5\" (UniqueName: \"kubernetes.io/projected/7af1b21a-e734-4f08-aab1-9d8308baeff6-kube-api-access-fz6l5\") pod \"7af1b21a-e734-4f08-aab1-9d8308baeff6\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") "
Nov 03 21:58:58.508503 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.508347    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-podinfo\") pod \"7af1b21a-e734-4f08-aab1-9d8308baeff6\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") "
Nov 03 21:58:58.508503 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.508394    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-workload-socket\") pod \"7af1b21a-e734-4f08-aab1-9d8308baeff6\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") "
Nov 03 21:58:58.508503 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.508427    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-workload-certs\") pod \"7af1b21a-e734-4f08-aab1-9d8308baeff6\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") "
Nov 03 21:58:58.508737 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.508530    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-token\" (UniqueName: \"kubernetes.io/projected/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-token\") pod \"7af1b21a-e734-4f08-aab1-9d8308baeff6\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") "
Nov 03 21:58:58.508737 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.508567    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-data\") pod \"7af1b21a-e734-4f08-aab1-9d8308baeff6\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") "
Nov 03 21:58:58.508737 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.508645    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7af1b21a-e734-4f08-aab1-9d8308baeff6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af1b21a-e734-4f08-aab1-9d8308baeff6-istiod-ca-cert\") pod \"7af1b21a-e734-4f08-aab1-9d8308baeff6\" (UID: \"7af1b21a-e734-4f08-aab1-9d8308baeff6\") "
Nov 03 21:58:58.508737 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.508658    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-workload-socket" pod "7af1b21a-e734-4f08-aab1-9d8308baeff6" (UID: "7af1b21a-e734-4f08-aab1-9d8308baeff6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:58:58.508737 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.508704    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-workload-certs" pod "7af1b21a-e734-4f08-aab1-9d8308baeff6" (UID: "7af1b21a-e734-4f08-aab1-9d8308baeff6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:58:58.509155 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.508851    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-data" pod "7af1b21a-e734-4f08-aab1-9d8308baeff6" (UID: "7af1b21a-e734-4f08-aab1-9d8308baeff6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:58:58.509216 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.509192    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-credential-socket" pod "7af1b21a-e734-4f08-aab1-9d8308baeff6" (UID: "7af1b21a-e734-4f08-aab1-9d8308baeff6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:58:58.509304 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.509218    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:58.509304 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.509277    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:58.509304 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.509293    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:58.509470 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.509367    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af1b21a-e734-4f08-aab1-9d8308baeff6-istiod-ca-cert" pod "7af1b21a-e734-4f08-aab1-9d8308baeff6" (UID: "7af1b21a-e734-4f08-aab1-9d8308baeff6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:58:58.510668 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.510644    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-envoy" pod "7af1b21a-e734-4f08-aab1-9d8308baeff6" (UID: "7af1b21a-e734-4f08-aab1-9d8308baeff6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:58:58.510838 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.510764    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-podinfo" pod "7af1b21a-e734-4f08-aab1-9d8308baeff6" (UID: "7af1b21a-e734-4f08-aab1-9d8308baeff6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:58:58.511208 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.511186    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af1b21a-e734-4f08-aab1-9d8308baeff6-kube-api-access-fz6l5" pod "7af1b21a-e734-4f08-aab1-9d8308baeff6" (UID: "7af1b21a-e734-4f08-aab1-9d8308baeff6"). InnerVolumeSpecName "kube-api-access-fz6l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:58:58.511307 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.511230    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-token" pod "7af1b21a-e734-4f08-aab1-9d8308baeff6" (UID: "7af1b21a-e734-4f08-aab1-9d8308baeff6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:58:58.610500 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.610467    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:58.610500 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.610505    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af1b21a-e734-4f08-aab1-9d8308baeff6-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:58.610857 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.610519    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:58.610857 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.610533    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:58.610857 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.610548    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz6l5\" (UniqueName: \"kubernetes.io/projected/7af1b21a-e734-4f08-aab1-9d8308baeff6-kube-api-access-fz6l5\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:58.610857 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.610562    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af1b21a-e734-4f08-aab1-9d8308baeff6-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:58.612391 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.612363    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl"
Nov 03 21:58:58.711533 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.711494    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7212b57-eddc-4cb8-8657-f3842ab5ad5d-kube-api-access-bp757\" (UniqueName: \"kubernetes.io/projected/f7212b57-eddc-4cb8-8657-f3842ab5ad5d-kube-api-access-bp757\") pod \"f7212b57-eddc-4cb8-8657-f3842ab5ad5d\" (UID: \"f7212b57-eddc-4cb8-8657-f3842ab5ad5d\") "
Nov 03 21:58:58.711719 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.711579    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7212b57-eddc-4cb8-8657-f3842ab5ad5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7212b57-eddc-4cb8-8657-f3842ab5ad5d-maas-api-xks-ca\") pod \"f7212b57-eddc-4cb8-8657-f3842ab5ad5d\" (UID: \"f7212b57-eddc-4cb8-8657-f3842ab5ad5d\") "
Nov 03 21:58:58.711719 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.711648    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7212b57-eddc-4cb8-8657-f3842ab5ad5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7212b57-eddc-4cb8-8657-f3842ab5ad5d-maas-api-openshift-ca\") pod \"f7212b57-eddc-4cb8-8657-f3842ab5ad5d\" (UID: \"f7212b57-eddc-4cb8-8657-f3842ab5ad5d\") "
Nov 03 21:58:58.712026 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.712001    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7212b57-eddc-4cb8-8657-f3842ab5ad5d-maas-api-openshift-ca" pod "f7212b57-eddc-4cb8-8657-f3842ab5ad5d" (UID: "f7212b57-eddc-4cb8-8657-f3842ab5ad5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:58:58.712230 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.712214    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7212b57-eddc-4cb8-8657-f3842ab5ad5d-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:58.713746 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.713718    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7212b57-eddc-4cb8-8657-f3842ab5ad5d-maas-api-xks-ca" pod "f7212b57-eddc-4cb8-8657-f3842ab5ad5d" (UID: "f7212b57-eddc-4cb8-8657-f3842ab5ad5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:58:58.713822 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.713775    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7212b57-eddc-4cb8-8657-f3842ab5ad5d-kube-api-access-bp757" pod "f7212b57-eddc-4cb8-8657-f3842ab5ad5d" (UID: "f7212b57-eddc-4cb8-8657-f3842ab5ad5d"). InnerVolumeSpecName "kube-api-access-bp757". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:58:58.813633 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.813596    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7212b57-eddc-4cb8-8657-f3842ab5ad5d-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:58.813633 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.813631    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp757\" (UniqueName: \"kubernetes.io/projected/f7212b57-eddc-4cb8-8657-f3842ab5ad5d-kube-api-access-bp757\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:58.877101 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.877050    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"]
Nov 03 21:58:58.880128 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:58.880099    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-292cf1-openshift-default-7b6697c9d9-fnfhw"]
Nov 03 21:58:59.462484 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.462460    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl"
Nov 03 21:58:59.462484 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.462469    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl" event={"ID":"f7212b57-eddc-4cb8-8657-f3842ab5ad5d","Type":"ContainerDied","Data":"8b335bedacbab619cd906d2e6e9b15e3ff5704b06492ed24105ee75440851bd2"}
Nov 03 21:58:59.462906 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.462510    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b335bedacbab619cd906d2e6e9b15e3ff5704b06492ed24105ee75440851bd2"
Nov 03 21:58:59.669543 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.669508    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7af1b21a-e734-4f08-aab1-9d8308baeff6" path="/var/lib/kubelet/pods/7af1b21a-e734-4f08-aab1-9d8308baeff6/volumes"
Nov 03 21:58:59.841438 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.841415    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:58:59.927583 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.927554    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac387b47-5c73-4984-bda9-8b5c82af02a5-kube-api-access-kjhjt\" (UniqueName: \"kubernetes.io/projected/ac387b47-5c73-4984-bda9-8b5c82af02a5-kube-api-access-kjhjt\") pod \"ac387b47-5c73-4984-bda9-8b5c82af02a5\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") "
Nov 03 21:58:59.927761 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.927648    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-workload-socket\") pod \"ac387b47-5c73-4984-bda9-8b5c82af02a5\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") "
Nov 03 21:58:59.927761 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.927713    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-podinfo\") pod \"ac387b47-5c73-4984-bda9-8b5c82af02a5\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") "
Nov 03 21:58:59.927841 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.927764    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac387b47-5c73-4984-bda9-8b5c82af02a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac387b47-5c73-4984-bda9-8b5c82af02a5-istiod-ca-cert\") pod \"ac387b47-5c73-4984-bda9-8b5c82af02a5\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") "
Nov 03 21:58:59.927841 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.927812    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-workload-certs\") pod \"ac387b47-5c73-4984-bda9-8b5c82af02a5\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") "
Nov 03 21:58:59.927929 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.927858    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-token\" (UniqueName: \"kubernetes.io/projected/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-token\") pod \"ac387b47-5c73-4984-bda9-8b5c82af02a5\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") "
Nov 03 21:58:59.927929 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.927889    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-envoy\") pod \"ac387b47-5c73-4984-bda9-8b5c82af02a5\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") "
Nov 03 21:58:59.927929 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.927919    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-credential-socket\") pod \"ac387b47-5c73-4984-bda9-8b5c82af02a5\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") "
Nov 03 21:58:59.928084 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.927936    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-workload-socket" pod "ac387b47-5c73-4984-bda9-8b5c82af02a5" (UID: "ac387b47-5c73-4984-bda9-8b5c82af02a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:58:59.928084 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.927951    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-data\") pod \"ac387b47-5c73-4984-bda9-8b5c82af02a5\" (UID: \"ac387b47-5c73-4984-bda9-8b5c82af02a5\") "
Nov 03 21:58:59.928278 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.928122    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-workload-certs" pod "ac387b47-5c73-4984-bda9-8b5c82af02a5" (UID: "ac387b47-5c73-4984-bda9-8b5c82af02a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:58:59.928278 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.928137    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac387b47-5c73-4984-bda9-8b5c82af02a5-istiod-ca-cert" pod "ac387b47-5c73-4984-bda9-8b5c82af02a5" (UID: "ac387b47-5c73-4984-bda9-8b5c82af02a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:58:59.928375 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.928319    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-credential-socket" pod "ac387b47-5c73-4984-bda9-8b5c82af02a5" (UID: "ac387b47-5c73-4984-bda9-8b5c82af02a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:58:59.928375 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.928365    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-data" pod "ac387b47-5c73-4984-bda9-8b5c82af02a5" (UID: "ac387b47-5c73-4984-bda9-8b5c82af02a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:58:59.928603 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.928586    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:59.928661 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.928606    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:59.928661 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.928615    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:59.928661 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.928624    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:59.928661 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.928636    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac387b47-5c73-4984-bda9-8b5c82af02a5-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:58:59.930109 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.930078    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-podinfo" pod "ac387b47-5c73-4984-bda9-8b5c82af02a5" (UID: "ac387b47-5c73-4984-bda9-8b5c82af02a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:58:59.930457 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.930437    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-token" pod "ac387b47-5c73-4984-bda9-8b5c82af02a5" (UID: "ac387b47-5c73-4984-bda9-8b5c82af02a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:58:59.930542 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.930525    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-envoy" pod "ac387b47-5c73-4984-bda9-8b5c82af02a5" (UID: "ac387b47-5c73-4984-bda9-8b5c82af02a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:58:59.930613 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:58:59.930552    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac387b47-5c73-4984-bda9-8b5c82af02a5-kube-api-access-kjhjt" pod "ac387b47-5c73-4984-bda9-8b5c82af02a5" (UID: "ac387b47-5c73-4984-bda9-8b5c82af02a5"). InnerVolumeSpecName "kube-api-access-kjhjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:59:00.030022 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.029930    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:59:00.030022 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.029964    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:59:00.030022 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.029978    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjhjt\" (UniqueName: \"kubernetes.io/projected/ac387b47-5c73-4984-bda9-8b5c82af02a5-kube-api-access-kjhjt\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:59:00.030022 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.029990    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac387b47-5c73-4984-bda9-8b5c82af02a5-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:59:00.471509 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.471468    5053 generic.go:356] "Generic (PLEG): container finished" podID="ac387b47-5c73-4984-bda9-8b5c82af02a5" containerID="68299c7f150ba045e58575e0ba430e41fe57525755a55899426289ee4192c0d7" exitCode=0
Nov 03 21:59:00.471509 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.471515    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"
Nov 03 21:59:00.472042 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.471578    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s" event={"ID":"ac387b47-5c73-4984-bda9-8b5c82af02a5","Type":"ContainerDied","Data":"68299c7f150ba045e58575e0ba430e41fe57525755a55899426289ee4192c0d7"}
Nov 03 21:59:00.472042 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.471632    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s" event={"ID":"ac387b47-5c73-4984-bda9-8b5c82af02a5","Type":"ContainerDied","Data":"38c567c611120be8785c27d8900c1cb0fae7dd89033250f0c840ac1050be3d53"}
Nov 03 21:59:00.472042 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.471660    5053 scope.go:122] "RemoveContainer" containerID="68299c7f150ba045e58575e0ba430e41fe57525755a55899426289ee4192c0d7"
Nov 03 21:59:00.483032 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.483012    5053 scope.go:122] "RemoveContainer" containerID="d6e43512cf8808bd2cb9564c7ec1505b947d81e3c8fccca628e536a11bf65372"
Nov 03 21:59:00.492612 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.492589    5053 scope.go:122] "RemoveContainer" containerID="68299c7f150ba045e58575e0ba430e41fe57525755a55899426289ee4192c0d7"
Nov 03 21:59:00.492848 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:59:00.492832    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68299c7f150ba045e58575e0ba430e41fe57525755a55899426289ee4192c0d7\": container with ID starting with 68299c7f150ba045e58575e0ba430e41fe57525755a55899426289ee4192c0d7 not found: ID does not exist" containerID="68299c7f150ba045e58575e0ba430e41fe57525755a55899426289ee4192c0d7"
Nov 03 21:59:00.492926 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.492857    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68299c7f150ba045e58575e0ba430e41fe57525755a55899426289ee4192c0d7"} err="failed to get container status \"68299c7f150ba045e58575e0ba430e41fe57525755a55899426289ee4192c0d7\": rpc error: code = NotFound desc = could not find container \"68299c7f150ba045e58575e0ba430e41fe57525755a55899426289ee4192c0d7\": container with ID starting with 68299c7f150ba045e58575e0ba430e41fe57525755a55899426289ee4192c0d7 not found: ID does not exist"
Nov 03 21:59:00.492926 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.492873    5053 scope.go:122] "RemoveContainer" containerID="d6e43512cf8808bd2cb9564c7ec1505b947d81e3c8fccca628e536a11bf65372"
Nov 03 21:59:00.493111 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:59:00.493095    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6e43512cf8808bd2cb9564c7ec1505b947d81e3c8fccca628e536a11bf65372\": container with ID starting with d6e43512cf8808bd2cb9564c7ec1505b947d81e3c8fccca628e536a11bf65372 not found: ID does not exist" containerID="d6e43512cf8808bd2cb9564c7ec1505b947d81e3c8fccca628e536a11bf65372"
Nov 03 21:59:00.493166 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.493114    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6e43512cf8808bd2cb9564c7ec1505b947d81e3c8fccca628e536a11bf65372"} err="failed to get container status \"d6e43512cf8808bd2cb9564c7ec1505b947d81e3c8fccca628e536a11bf65372\": rpc error: code = NotFound desc = could not find container \"d6e43512cf8808bd2cb9564c7ec1505b947d81e3c8fccca628e536a11bf65372\": container with ID starting with d6e43512cf8808bd2cb9564c7ec1505b947d81e3c8fccca628e536a11bf65372 not found: ID does not exist"
Nov 03 21:59:00.635602 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.635572    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"]
Nov 03 21:59:00.685122 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.685093    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c993b8-openshift-default-5ffb55f549-2tg4s"]
Nov 03 21:59:00.964790 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.964684    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74196013-crh2p"]
Nov 03 21:59:00.965378 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.965356    5053 cpu_manager.go:414] "containerMap: removing container" podUID="7af1b21a-e734-4f08-aab1-9d8308baeff6" containerName="istio-proxy"
Nov 03 21:59:00.965378 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.965376    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7af1b21a-e734-4f08-aab1-9d8308baeff6" containerName="istio-proxy"
Nov 03 21:59:00.965546 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.965395    5053 cpu_manager.go:414] "containerMap: removing container" podUID="ac387b47-5c73-4984-bda9-8b5c82af02a5" containerName="istio-proxy"
Nov 03 21:59:00.965546 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.965402    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac387b47-5c73-4984-bda9-8b5c82af02a5" containerName="istio-proxy"
Nov 03 21:59:00.965546 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.965413    5053 cpu_manager.go:414] "containerMap: removing container" podUID="f7212b57-eddc-4cb8-8657-f3842ab5ad5d" containerName="revoke-keys"
Nov 03 21:59:00.965546 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.965419    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7212b57-eddc-4cb8-8657-f3842ab5ad5d" containerName="revoke-keys"
Nov 03 21:59:00.965546 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.965430    5053 cpu_manager.go:414] "containerMap: removing container" podUID="7af1b21a-e734-4f08-aab1-9d8308baeff6" containerName="istio-proxy"
Nov 03 21:59:00.965546 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.965434    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7af1b21a-e734-4f08-aab1-9d8308baeff6" containerName="istio-proxy"
Nov 03 21:59:00.965546 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.965532    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="7af1b21a-e734-4f08-aab1-9d8308baeff6" containerName="istio-proxy"
Nov 03 21:59:00.965546 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.965542    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac387b47-5c73-4984-bda9-8b5c82af02a5" containerName="istio-proxy"
Nov 03 21:59:00.965833 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.965553    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac387b47-5c73-4984-bda9-8b5c82af02a5" containerName="istio-proxy"
Nov 03 21:59:00.965833 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.965559    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7212b57-eddc-4cb8-8657-f3842ab5ad5d" containerName="revoke-keys"
Nov 03 21:59:00.970460 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:00.970437    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74196013-crh2p"
Nov 03 21:59:01.023298 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:01.022385    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:59:01.023298 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:01.022510    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:59:01.024419 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:01.024348    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74196013-crh2p"]
Nov 03 21:59:01.146601 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:01.146559    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c4fa70d-609b-481d-ae55-790ca51dd22b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74196013-crh2p\" (UID: \"3c4fa70d-609b-481d-ae55-790ca51dd22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74196013-crh2p"
Nov 03 21:59:01.146822 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:01.146705    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcw9w\" (UniqueName: \"kubernetes.io/projected/3c4fa70d-609b-481d-ae55-790ca51dd22b-kube-api-access-kcw9w\") pod \"maas-api-revoke-sub-74196013-crh2p\" (UID: \"3c4fa70d-609b-481d-ae55-790ca51dd22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74196013-crh2p"
Nov 03 21:59:01.146894 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:01.146865    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c4fa70d-609b-481d-ae55-790ca51dd22b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74196013-crh2p\" (UID: \"3c4fa70d-609b-481d-ae55-790ca51dd22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74196013-crh2p"
Nov 03 21:59:01.248261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:01.248134    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c4fa70d-609b-481d-ae55-790ca51dd22b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74196013-crh2p\" (UID: \"3c4fa70d-609b-481d-ae55-790ca51dd22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74196013-crh2p"
Nov 03 21:59:01.248261 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:01.248210    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c4fa70d-609b-481d-ae55-790ca51dd22b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74196013-crh2p\" (UID: \"3c4fa70d-609b-481d-ae55-790ca51dd22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74196013-crh2p"
Nov 03 21:59:01.248510 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:01.248336    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcw9w\" (UniqueName: \"kubernetes.io/projected/3c4fa70d-609b-481d-ae55-790ca51dd22b-kube-api-access-kcw9w\") pod \"maas-api-revoke-sub-74196013-crh2p\" (UID: \"3c4fa70d-609b-481d-ae55-790ca51dd22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74196013-crh2p"
Nov 03 21:59:01.248895 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:01.248873    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c4fa70d-609b-481d-ae55-790ca51dd22b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74196013-crh2p\" (UID: \"3c4fa70d-609b-481d-ae55-790ca51dd22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74196013-crh2p"
Nov 03 21:59:01.250780 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:01.250754    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c4fa70d-609b-481d-ae55-790ca51dd22b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74196013-crh2p\" (UID: \"3c4fa70d-609b-481d-ae55-790ca51dd22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74196013-crh2p"
Nov 03 21:59:01.305175 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:01.305135    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcw9w\" (UniqueName: \"kubernetes.io/projected/3c4fa70d-609b-481d-ae55-790ca51dd22b-kube-api-access-kcw9w\") pod \"maas-api-revoke-sub-74196013-crh2p\" (UID: \"3c4fa70d-609b-481d-ae55-790ca51dd22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74196013-crh2p"
Nov 03 21:59:01.586706 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:01.586618    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74196013-crh2p"
Nov 03 21:59:01.673131 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:01.673097    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac387b47-5c73-4984-bda9-8b5c82af02a5" path="/var/lib/kubelet/pods/ac387b47-5c73-4984-bda9-8b5c82af02a5/volumes"
Nov 03 21:59:01.970466 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:59:01.970421    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4fa70d_609b_481d_ae55_790ca51dd22b.slice/crio-35ee189441ff5b68ba4141f99b4a17f2933ef59c0a3fd882fe3cdf9a406e164c WatchSource:0}: Error finding container 35ee189441ff5b68ba4141f99b4a17f2933ef59c0a3fd882fe3cdf9a406e164c: Status 404 returned error can't find the container with id 35ee189441ff5b68ba4141f99b4a17f2933ef59c0a3fd882fe3cdf9a406e164c
Nov 03 21:59:01.970862 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:01.970834    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74196013-crh2p"]
Nov 03 21:59:02.488018 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:02.487973    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74196013-crh2p" event={"ID":"3c4fa70d-609b-481d-ae55-790ca51dd22b","Type":"ContainerStarted","Data":"747b2dcacce878c5ac6a4d24e4a08e489e1dadd5e0d7f0492be4923ae0ff506c"}
Nov 03 21:59:02.488198 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:02.488037    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74196013-crh2p" event={"ID":"3c4fa70d-609b-481d-ae55-790ca51dd22b","Type":"ContainerStarted","Data":"35ee189441ff5b68ba4141f99b4a17f2933ef59c0a3fd882fe3cdf9a406e164c"}
Nov 03 21:59:02.607019 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:02.606919    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74196013-crh2p" podStartSLOduration=2.6069031000000003 podStartE2EDuration="2.6069031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:02.60392555 +0000 UTC m=+2657.097847426" watchObservedRunningTime="2026-11-03 21:59:02.6069031 +0000 UTC m=+2657.100824974"
Nov 03 21:59:03.496616 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:03.496582    5053 generic.go:356] "Generic (PLEG): container finished" podID="3c4fa70d-609b-481d-ae55-790ca51dd22b" containerID="747b2dcacce878c5ac6a4d24e4a08e489e1dadd5e0d7f0492be4923ae0ff506c" exitCode=0
Nov 03 21:59:03.496827 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:03.496660    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74196013-crh2p" event={"ID":"3c4fa70d-609b-481d-ae55-790ca51dd22b","Type":"ContainerDied","Data":"747b2dcacce878c5ac6a4d24e4a08e489e1dadd5e0d7f0492be4923ae0ff506c"}
Nov 03 21:59:04.675065 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:04.675040    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74196013-crh2p"
Nov 03 21:59:04.682921 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:04.682900    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c4fa70d-609b-481d-ae55-790ca51dd22b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c4fa70d-609b-481d-ae55-790ca51dd22b-maas-api-xks-ca\") pod \"3c4fa70d-609b-481d-ae55-790ca51dd22b\" (UID: \"3c4fa70d-609b-481d-ae55-790ca51dd22b\") "
Nov 03 21:59:04.683057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:04.682952    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4fa70d-609b-481d-ae55-790ca51dd22b-kube-api-access-kcw9w\" (UniqueName: \"kubernetes.io/projected/3c4fa70d-609b-481d-ae55-790ca51dd22b-kube-api-access-kcw9w\") pod \"3c4fa70d-609b-481d-ae55-790ca51dd22b\" (UID: \"3c4fa70d-609b-481d-ae55-790ca51dd22b\") "
Nov 03 21:59:04.683057 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:04.682992    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c4fa70d-609b-481d-ae55-790ca51dd22b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c4fa70d-609b-481d-ae55-790ca51dd22b-maas-api-openshift-ca\") pod \"3c4fa70d-609b-481d-ae55-790ca51dd22b\" (UID: \"3c4fa70d-609b-481d-ae55-790ca51dd22b\") "
Nov 03 21:59:04.683503 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:04.683474    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c4fa70d-609b-481d-ae55-790ca51dd22b-maas-api-openshift-ca" pod "3c4fa70d-609b-481d-ae55-790ca51dd22b" (UID: "3c4fa70d-609b-481d-ae55-790ca51dd22b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:59:04.685169 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:04.685143    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4fa70d-609b-481d-ae55-790ca51dd22b-kube-api-access-kcw9w" pod "3c4fa70d-609b-481d-ae55-790ca51dd22b" (UID: "3c4fa70d-609b-481d-ae55-790ca51dd22b"). InnerVolumeSpecName "kube-api-access-kcw9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:59:04.685169 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:04.685156    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c4fa70d-609b-481d-ae55-790ca51dd22b-maas-api-xks-ca" pod "3c4fa70d-609b-481d-ae55-790ca51dd22b" (UID: "3c4fa70d-609b-481d-ae55-790ca51dd22b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:59:04.785469 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:04.785378    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcw9w\" (UniqueName: \"kubernetes.io/projected/3c4fa70d-609b-481d-ae55-790ca51dd22b-kube-api-access-kcw9w\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:59:04.785469 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:04.785421    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c4fa70d-609b-481d-ae55-790ca51dd22b-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:59:04.785469 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:04.785437    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c4fa70d-609b-481d-ae55-790ca51dd22b-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:59:05.511714 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:05.511679    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74196013-crh2p"
Nov 03 21:59:05.511714 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:05.511707    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74196013-crh2p" event={"ID":"3c4fa70d-609b-481d-ae55-790ca51dd22b","Type":"ContainerDied","Data":"35ee189441ff5b68ba4141f99b4a17f2933ef59c0a3fd882fe3cdf9a406e164c"}
Nov 03 21:59:05.511983 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:05.511756    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35ee189441ff5b68ba4141f99b4a17f2933ef59c0a3fd882fe3cdf9a406e164c"
Nov 03 21:59:10.953136 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:59:10.953084    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:59:12.682639 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:12.682607    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"]
Nov 03 21:59:12.683352 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:12.683333    5053 cpu_manager.go:414] "containerMap: removing container" podUID="3c4fa70d-609b-481d-ae55-790ca51dd22b" containerName="revoke-keys"
Nov 03 21:59:12.683352 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:12.683353    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c4fa70d-609b-481d-ae55-790ca51dd22b" containerName="revoke-keys"
Nov 03 21:59:12.683480 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:12.683399    5053 cpu_manager.go:414] "containerMap: removing container" podUID="ac387b47-5c73-4984-bda9-8b5c82af02a5" containerName="istio-proxy"
Nov 03 21:59:12.683480 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:12.683404    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac387b47-5c73-4984-bda9-8b5c82af02a5" containerName="istio-proxy"
Nov 03 21:59:12.683545 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:12.683506    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="7af1b21a-e734-4f08-aab1-9d8308baeff6" containerName="istio-proxy"
Nov 03 21:59:12.683545 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:12.683520    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c4fa70d-609b-481d-ae55-790ca51dd22b" containerName="revoke-keys"
Nov 03 21:59:12.686901 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:12.686876    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"
Nov 03 21:59:12.739059 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:12.739026    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-4644ae-serving-cert\""
Nov 03 21:59:12.788303 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:12.788272    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/604c02df-5b27-4ed4-9e65-a32f6f7207d0-metrics-certs\") pod \"maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6\" (UID: \"604c02df-5b27-4ed4-9e65-a32f6f7207d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"
Nov 03 21:59:12.788465 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:12.788374    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604c02df-5b27-4ed4-9e65-a32f6f7207d0-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6\" (UID: \"604c02df-5b27-4ed4-9e65-a32f6f7207d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"
Nov 03 21:59:12.788570 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:12.788473    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604c02df-5b27-4ed4-9e65-a32f6f7207d0-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6\" (UID: \"604c02df-5b27-4ed4-9e65-a32f6f7207d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"
Nov 03 21:59:12.788570 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:12.788533    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr8r5\" (UniqueName: \"kubernetes.io/projected/604c02df-5b27-4ed4-9e65-a32f6f7207d0-kube-api-access-nr8r5\") pod \"maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6\" (UID: \"604c02df-5b27-4ed4-9e65-a32f6f7207d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"
Nov 03 21:59:12.791562 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:12.791538    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"]
Nov 03 21:59:12.889638 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:12.889604    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/604c02df-5b27-4ed4-9e65-a32f6f7207d0-metrics-certs\") pod \"maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6\" (UID: \"604c02df-5b27-4ed4-9e65-a32f6f7207d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"
Nov 03 21:59:12.889814 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:12.889666    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604c02df-5b27-4ed4-9e65-a32f6f7207d0-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6\" (UID: \"604c02df-5b27-4ed4-9e65-a32f6f7207d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"
Nov 03 21:59:12.889814 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:12.889718    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604c02df-5b27-4ed4-9e65-a32f6f7207d0-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6\" (UID: \"604c02df-5b27-4ed4-9e65-a32f6f7207d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"
Nov 03 21:59:12.889814 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:12.889758    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr8r5\" (UniqueName: \"kubernetes.io/projected/604c02df-5b27-4ed4-9e65-a32f6f7207d0-kube-api-access-nr8r5\") pod \"maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6\" (UID: \"604c02df-5b27-4ed4-9e65-a32f6f7207d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"
Nov 03 21:59:12.889935 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:59:12.889806    5053 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-serving-cert: secret "maas-api-e2e-praxis-a-4644ae-serving-cert" not found
Nov 03 21:59:12.889935 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:59:12.889870    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/604c02df-5b27-4ed4-9e65-a32f6f7207d0-maas-api-tls podName:604c02df-5b27-4ed4-9e65-a32f6f7207d0 nodeName:}" failed. No retries permitted until 2026-11-03 21:59:13.389852811 +0000 UTC m=+2667.883774678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/604c02df-5b27-4ed4-9e65-a32f6f7207d0-maas-api-tls") pod "maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6" (UID: "604c02df-5b27-4ed4-9e65-a32f6f7207d0") : secret "maas-api-e2e-praxis-a-4644ae-serving-cert" not found
Nov 03 21:59:12.890410 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:12.890388    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604c02df-5b27-4ed4-9e65-a32f6f7207d0-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6\" (UID: \"604c02df-5b27-4ed4-9e65-a32f6f7207d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"
Nov 03 21:59:12.892400 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:12.892378    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/604c02df-5b27-4ed4-9e65-a32f6f7207d0-metrics-certs\") pod \"maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6\" (UID: \"604c02df-5b27-4ed4-9e65-a32f6f7207d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"
Nov 03 21:59:12.953711 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:12.953674    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr8r5\" (UniqueName: \"kubernetes.io/projected/604c02df-5b27-4ed4-9e65-a32f6f7207d0-kube-api-access-nr8r5\") pod \"maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6\" (UID: \"604c02df-5b27-4ed4-9e65-a32f6f7207d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"
Nov 03 21:59:13.396650 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:13.396569    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604c02df-5b27-4ed4-9e65-a32f6f7207d0-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6\" (UID: \"604c02df-5b27-4ed4-9e65-a32f6f7207d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"
Nov 03 21:59:13.399152 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:13.399129    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604c02df-5b27-4ed4-9e65-a32f6f7207d0-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6\" (UID: \"604c02df-5b27-4ed4-9e65-a32f6f7207d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"
Nov 03 21:59:13.601520 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:13.601486    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"
Nov 03 21:59:14.029296 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:14.027630    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"]
Nov 03 21:59:14.034834 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:59:14.034796    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604c02df_5b27_4ed4_9e65_a32f6f7207d0.slice/crio-09e6f1efa51d377a0408c09afe38ca5eb09bffe192ccb0811ce5c07cbc22f5d8 WatchSource:0}: Error finding container 09e6f1efa51d377a0408c09afe38ca5eb09bffe192ccb0811ce5c07cbc22f5d8: Status 404 returned error can't find the container with id 09e6f1efa51d377a0408c09afe38ca5eb09bffe192ccb0811ce5c07cbc22f5d8
Nov 03 21:59:14.572126 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:14.572091    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6" event={"ID":"604c02df-5b27-4ed4-9e65-a32f6f7207d0","Type":"ContainerStarted","Data":"09e6f1efa51d377a0408c09afe38ca5eb09bffe192ccb0811ce5c07cbc22f5d8"}
Nov 03 21:59:15.581814 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:15.581778    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6" event={"ID":"604c02df-5b27-4ed4-9e65-a32f6f7207d0","Type":"ContainerStarted","Data":"ddc0cacf5da1f00985937feb3c265797c8ac2031152aa27fa2eb6534c7bd5256"}
Nov 03 21:59:15.582203 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:15.581902    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"
Nov 03 21:59:15.698750 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:15.698698    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6" podStartSLOduration=2.859672525 podStartE2EDuration="3.698684115s" totalImagesPullingTime="839.01159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:15.697322198 +0000 UTC m=+2670.191244074" watchObservedRunningTime="2026-11-03 21:59:15.698684115 +0000 UTC m=+2670.192605989"
Nov 03 21:59:19.612376 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:19.612342    5053 generic.go:356] "Generic (PLEG): container finished" podID="122b09a6-7732-417d-b82f-a977510c180f" containerID="292948e4406f710ebace8840ca6143130122c4774da6958deb001b408cc68ebe" exitCode=0
Nov 03 21:59:19.612804 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:19.612408    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd" event={"ID":"122b09a6-7732-417d-b82f-a977510c180f","Type":"ContainerDied","Data":"292948e4406f710ebace8840ca6143130122c4774da6958deb001b408cc68ebe"}
Nov 03 21:59:19.612848 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:19.612818    5053 scope.go:122] "RemoveContainer" containerID="292948e4406f710ebace8840ca6143130122c4774da6958deb001b408cc68ebe"
Nov 03 21:59:19.614298 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:19.614271    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:59:19.614357 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:19.614344    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:59:19.614408 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:19.614393    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:59:20.638208 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:20.638166    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd" event={"ID":"122b09a6-7732-417d-b82f-a977510c180f","Type":"ContainerStarted","Data":"6b75ded44a8b59d2f9c9a809f315c0372e4dc771a8b3f6c8025de9314e90e473"}
Nov 03 21:59:20.893504 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:20.893416    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:59:20.894964 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:20.894940    5053 patch_prober.go:29] interesting pod/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 03 21:59:20.895032 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:20.894986    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd" podUID="122b09a6-7732-417d-b82f-a977510c180f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 03 21:59:21.594536 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:21.594510    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"
Nov 03 21:59:21.693176 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:21.693138    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:59:21.893137 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:21.893057    5053 patch_prober.go:29] interesting pod/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 03 21:59:21.893137 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:21.893110    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd" podUID="122b09a6-7732-417d-b82f-a977510c180f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 03 21:59:22.894007 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:22.893974    5053 patch_prober.go:29] interesting pod/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 03 21:59:22.894443 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:22.894032    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd" podUID="122b09a6-7732-417d-b82f-a977510c180f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 03 21:59:23.893832 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:23.893794    5053 patch_prober.go:29] interesting pod/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 03 21:59:23.894027 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:23.893860    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd" podUID="122b09a6-7732-417d-b82f-a977510c180f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 03 21:59:24.894006 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:24.893974    5053 patch_prober.go:29] interesting pod/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 03 21:59:24.894200 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:24.894024    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd" podUID="122b09a6-7732-417d-b82f-a977510c180f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 03 21:59:25.898649 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:25.898622    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:59:25.899742 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:25.899721    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 21:59:27.700750 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.700711    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"]
Nov 03 21:59:27.705470 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.705446    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.807329 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.807296    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"]
Nov 03 21:59:27.810141 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.810111    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-token\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.810332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.810179    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vdjv\" (UniqueName: \"kubernetes.io/projected/b8574f16-35c1-4717-a83a-396210ca7a1f-kube-api-access-8vdjv\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.810332 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.810290    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-envoy\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.810480 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.810378    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8574f16-35c1-4717-a83a-396210ca7a1f-istiod-ca-cert\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.810480 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.810417    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-credential-socket\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.810480 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.810455    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-workload-certs\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.810628 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.810485    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-podinfo\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.810628 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.810589    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-workload-socket\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.810744 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.810643    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-data\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.864291 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.862617    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-7990bd-openshift-default-dockercfg-vbbz8\""
Nov 03 21:59:27.912358 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.912323    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-credential-socket\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.912583 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.912382    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-workload-certs\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.912583 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.912416    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-podinfo\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.912583 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.912504    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-workload-socket\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.912583 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.912550    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-data\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.912800 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.912635    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-token\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.912800 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.912691    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vdjv\" (UniqueName: \"kubernetes.io/projected/b8574f16-35c1-4717-a83a-396210ca7a1f-kube-api-access-8vdjv\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.912800 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.912722    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-envoy\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.912951 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.912801    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8574f16-35c1-4717-a83a-396210ca7a1f-istiod-ca-cert\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.913541 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.913519    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8574f16-35c1-4717-a83a-396210ca7a1f-istiod-ca-cert\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.913795 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.913776    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-credential-socket\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.914007 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.913991    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-workload-certs\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.925126 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.915542    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-workload-socket\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.925126 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.915908    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-data\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.925126 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.918310    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-podinfo\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.925126 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.919633    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-envoy\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.977856 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.977779    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-token\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:27.977856 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:27.977829    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vdjv\" (UniqueName: \"kubernetes.io/projected/b8574f16-35c1-4717-a83a-396210ca7a1f-kube-api-access-8vdjv\") pod \"e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") " pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:28.026687 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:28.026652    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:28.454477 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:28.454450    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"]
Nov 03 21:59:28.455168 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:59:28.455136    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8574f16_35c1_4717_a83a_396210ca7a1f.slice/crio-ec38d20fe2f89aa2d39c2add1c554c4df5ddc25b422f7406e3b8bbc15458f8d9 WatchSource:0}: Error finding container ec38d20fe2f89aa2d39c2add1c554c4df5ddc25b422f7406e3b8bbc15458f8d9: Status 404 returned error can't find the container with id ec38d20fe2f89aa2d39c2add1c554c4df5ddc25b422f7406e3b8bbc15458f8d9
Nov 03 21:59:28.457677 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:28.457645    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:59:28.457760 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:28.457709    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:59:28.457760 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:28.457740    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:59:28.699413 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:28.699379    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g" event={"ID":"b8574f16-35c1-4717-a83a-396210ca7a1f","Type":"ContainerStarted","Data":"78cddba9c25d4cf6bb570f4c9e894c3c520919fe645ef8a1eca28f9e45fe8ed1"}
Nov 03 21:59:28.699627 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:28.699434    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g" event={"ID":"b8574f16-35c1-4717-a83a-396210ca7a1f","Type":"ContainerStarted","Data":"ec38d20fe2f89aa2d39c2add1c554c4df5ddc25b422f7406e3b8bbc15458f8d9"}
Nov 03 21:59:28.817728 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:28.817618    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g" podStartSLOduration=1.817603466 podStartE2EDuration="1.817603466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:28.814485558 +0000 UTC m=+2683.308407433" watchObservedRunningTime="2026-11-03 21:59:28.817603466 +0000 UTC m=+2683.311525340"
Nov 03 21:59:29.027096 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:29.027058    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:29.032860 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:29.032834    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:29.705530 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:29.705500    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:29.706458 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:29.706437    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 21:59:29.916026 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:29.915964    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"]
Nov 03 21:59:29.920360 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:29.920337    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:29.974585 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:29.973735    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-6de8e6-openshift-default-dockercfg-vxss9\""
Nov 03 21:59:29.978138 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:29.978112    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"]
Nov 03 21:59:30.040873 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.040835    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a54168-03bd-4b47-9c62-a9de6168027b-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.041068 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.040970    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.041138 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.041091    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.041198 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.041184    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.041274 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.041247    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.041344 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.041305    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.041344 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.041336    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.041459 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.041445    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf62w\" (UniqueName: \"kubernetes.io/projected/b5a54168-03bd-4b47-9c62-a9de6168027b-kube-api-access-hf62w\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.041553 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.041493    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.142438 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.142406    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.142438 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.142443    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.142733 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.142475    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.142733 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.142504    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.142733 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.142553    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf62w\" (UniqueName: \"kubernetes.io/projected/b5a54168-03bd-4b47-9c62-a9de6168027b-kube-api-access-hf62w\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.142733 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.142592    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.142733 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.142679    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a54168-03bd-4b47-9c62-a9de6168027b-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.142733 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.142733    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.143088 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.142822    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.143088 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.142960    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.143199 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.143124    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.143282 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.143206    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.143462 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.143434    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a54168-03bd-4b47-9c62-a9de6168027b-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.143462 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.143459    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.145286 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.145264    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.145412 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.145226    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.199104 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.199067    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.199322 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.199077    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf62w\" (UniqueName: \"kubernetes.io/projected/b5a54168-03bd-4b47-9c62-a9de6168027b-kube-api-access-hf62w\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.242929 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.242848    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:30.681327 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.681223    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"]
Nov 03 21:59:30.712588 ip-10-0-128-43 kubenswrapper[5053]: W1103 21:59:30.712551    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a54168_03bd_4b47_9c62_a9de6168027b.slice/crio-153ceb77324c12ba17131cfb9624e034a4553f80a94ff4ca5ae178f35b6b3300 WatchSource:0}: Error finding container 153ceb77324c12ba17131cfb9624e034a4553f80a94ff4ca5ae178f35b6b3300: Status 404 returned error can't find the container with id 153ceb77324c12ba17131cfb9624e034a4553f80a94ff4ca5ae178f35b6b3300
Nov 03 21:59:30.715721 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.715683    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:59:30.715796 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.715777    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:59:30.715830 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:30.715816    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:59:31.735171 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:31.735124    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9" event={"ID":"b5a54168-03bd-4b47-9c62-a9de6168027b","Type":"ContainerStarted","Data":"1f7152660eeb2fa9cdaa13784b3cab4d96a6d3eb4f23418168d58c2f9382b994"}
Nov 03 21:59:31.735680 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:31.735193    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9" event={"ID":"b5a54168-03bd-4b47-9c62-a9de6168027b","Type":"ContainerStarted","Data":"153ceb77324c12ba17131cfb9624e034a4553f80a94ff4ca5ae178f35b6b3300"}
Nov 03 21:59:31.965980 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:31.965930    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9" podStartSLOduration=2.965915747 podStartE2EDuration="2.965915747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:31.914715484 +0000 UTC m=+2686.408637362" watchObservedRunningTime="2026-11-03 21:59:31.965915747 +0000 UTC m=+2686.459837621"
Nov 03 21:59:32.243315 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:32.243279    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:32.244876 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:32.244852    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 03 21:59:32.245001 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:32.244907    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9" podUID="b5a54168-03bd-4b47-9c62-a9de6168027b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 03 21:59:33.243462 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:33.243424    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 03 21:59:33.243882 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:33.243494    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9" podUID="b5a54168-03bd-4b47-9c62-a9de6168027b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 03 21:59:34.243487 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:34.243452    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 03 21:59:34.243993 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:34.243519    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9" podUID="b5a54168-03bd-4b47-9c62-a9de6168027b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 03 21:59:35.244102 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:35.244065    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 03 21:59:35.244628 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:35.244126    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9" podUID="b5a54168-03bd-4b47-9c62-a9de6168027b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 03 21:59:36.243289 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:36.243254    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 03 21:59:36.243518 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:36.243327    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9" podUID="b5a54168-03bd-4b47-9c62-a9de6168027b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 03 21:59:37.243276 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:37.243219    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 03 21:59:37.243932 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:37.243880    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9" podUID="b5a54168-03bd-4b47-9c62-a9de6168027b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 03 21:59:38.248616 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:38.248586    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:38.249104 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:38.249043    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:38.250006 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:38.249982    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 21:59:40.963599 ip-10-0-128-43 kubenswrapper[5053]: E1103 21:59:40.963557    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 21:59:46.129652 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:46.129619    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 21:59:46.139067 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:46.139042    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 21:59:46.145703 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:46.145679    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:59:46.149585 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:46.149564    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 21:59:47.578865 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:47.578836    5053 scope.go:122] "RemoveContainer" containerID="bfa56daebeab5a1141372fcac9490bfeb98dbf61cab0514047e94d6d8aec08fd"
Nov 03 21:59:50.629841 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:50.629472    5053 patch_prober.go:29] interesting pod/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": read tcp 10.132.0.2:55514->10.132.0.198:15021: read: connection reset by peer" start-of-body=
Nov 03 21:59:50.629841 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:50.629553    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk" podUID="720e4837-f945-4df0-a637-c2dc41608b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": read tcp 10.132.0.2:55514->10.132.0.198:15021: read: connection reset by peer"
Nov 03 21:59:50.915137 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:50.915030    5053 generic.go:356] "Generic (PLEG): container finished" podID="720e4837-f945-4df0-a637-c2dc41608b7e" containerID="1f4e9cb903c04d892ac73fdfb0537b6103928248909ed4aa97b5c97b7367f471" exitCode=0
Nov 03 21:59:50.915137 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:50.915117    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk" event={"ID":"720e4837-f945-4df0-a637-c2dc41608b7e","Type":"ContainerDied","Data":"1f4e9cb903c04d892ac73fdfb0537b6103928248909ed4aa97b5c97b7367f471"}
Nov 03 21:59:50.915691 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:50.915670    5053 scope.go:122] "RemoveContainer" containerID="1f4e9cb903c04d892ac73fdfb0537b6103928248909ed4aa97b5c97b7367f471"
Nov 03 21:59:50.917325 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:50.917286    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:59:50.917415 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:50.917365    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:59:50.917415 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:50.917399    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:59:51.933614 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:51.933577    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk" event={"ID":"720e4837-f945-4df0-a637-c2dc41608b7e","Type":"ContainerStarted","Data":"90592ab9d76ada256f68841bce1fb4c6b331c10080d57e0fe48b853674cb4fc9"}
Nov 03 21:59:52.332868 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:52.332831    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:59:52.333101 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:52.333076    5053 patch_prober.go:29] interesting pod/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Nov 03 21:59:52.333153 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:52.333129    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk" podUID="720e4837-f945-4df0-a637-c2dc41608b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Nov 03 21:59:53.333801 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:53.333759    5053 patch_prober.go:29] interesting pod/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Nov 03 21:59:53.334269 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:53.333841    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk" podUID="720e4837-f945-4df0-a637-c2dc41608b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Nov 03 21:59:54.333604 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:54.333568    5053 patch_prober.go:29] interesting pod/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Nov 03 21:59:54.333794 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:54.333624    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk" podUID="720e4837-f945-4df0-a637-c2dc41608b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Nov 03 21:59:55.333826 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:55.333773    5053 patch_prober.go:29] interesting pod/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Nov 03 21:59:55.333826 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:55.333825    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk" podUID="720e4837-f945-4df0-a637-c2dc41608b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Nov 03 21:59:56.336779 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:56.336733    5053 patch_prober.go:29] interesting pod/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Nov 03 21:59:56.337197 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:56.336804    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk" podUID="720e4837-f945-4df0-a637-c2dc41608b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Nov 03 21:59:57.337364 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:57.337334    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:59:57.337966 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:57.337940    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 21:59:57.338466 ip-10-0-128-43 kubenswrapper[5053]: I1103 21:59:57.338450    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 22:00:00.166102 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.166063    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst"]
Nov 03 22:00:00.170501 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.170481    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx"]
Nov 03 22:00:00.170618 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.170598    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst"
Nov 03 22:00:00.174435 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.174413    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx"
Nov 03 22:00:00.189621 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.189599    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx"]
Nov 03 22:00:00.194032 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.194013    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx"
Nov 03 22:00:00.215892 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.215864    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z"]
Nov 03 22:00:00.219854 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.219832    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l"]
Nov 03 22:00:00.219978 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.219954    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z"
Nov 03 22:00:00.223824 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.223806    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p"]
Nov 03 22:00:00.223930 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.223911    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l"
Nov 03 22:00:00.227955 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.227939    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p"
Nov 03 22:00:00.265995 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.265963    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx"]
Nov 03 22:00:00.268969 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.268946    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst"]
Nov 03 22:00:00.286295 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.286266    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4vd4\" (UniqueName: \"kubernetes.io/projected/1999e701-573c-454b-baca-bf466f9c2ced-kube-api-access-b4vd4\") pod \"maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx\" (UID: \"1999e701-573c-454b-baca-bf466f9c2ced\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx"
Nov 03 22:00:00.286430 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.286302    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc9jd\" (UniqueName: \"kubernetes.io/projected/53bee6ec-657e-49b0-9e45-2b1c4c1b275f-kube-api-access-rc9jd\") pod \"maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst\" (UID: \"53bee6ec-657e-49b0-9e45-2b1c4c1b275f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst"
Nov 03 22:00:00.286639 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.286547    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc6ade3-0fe6-45d5-a8a1-36118b848724-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z\" (UID: \"bbc6ade3-0fe6-45d5-a8a1-36118b848724\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z"
Nov 03 22:00:00.286639 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.286600    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlr7w\" (UniqueName: \"kubernetes.io/projected/bbc6ade3-0fe6-45d5-a8a1-36118b848724-kube-api-access-tlr7w\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z\" (UID: \"bbc6ade3-0fe6-45d5-a8a1-36118b848724\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z"
Nov 03 22:00:00.286797 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.286739    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07b51d2-c47b-400a-b786-4d500d736169-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx\" (UID: \"f07b51d2-c47b-400a-b786-4d500d736169\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx"
Nov 03 22:00:00.286797 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.286785    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53bee6ec-657e-49b0-9e45-2b1c4c1b275f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst\" (UID: \"53bee6ec-657e-49b0-9e45-2b1c4c1b275f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst"
Nov 03 22:00:00.286909 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.286876    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1999e701-573c-454b-baca-bf466f9c2ced-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx\" (UID: \"1999e701-573c-454b-baca-bf466f9c2ced\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx"
Nov 03 22:00:00.286958 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.286910    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bdvt\" (UniqueName: \"kubernetes.io/projected/f07b51d2-c47b-400a-b786-4d500d736169-kube-api-access-6bdvt\") pod \"maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx\" (UID: \"f07b51d2-c47b-400a-b786-4d500d736169\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx"
Nov 03 22:00:00.326669 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.322493    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z"]
Nov 03 22:00:00.371819 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.371790    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:00:00.376540 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.376504    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx"]
Nov 03 22:00:00.379616 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.379591    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l"]
Nov 03 22:00:00.390107 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.390054    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc6ade3-0fe6-45d5-a8a1-36118b848724-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z\" (UID: \"bbc6ade3-0fe6-45d5-a8a1-36118b848724\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z"
Nov 03 22:00:00.390288 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.390129    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d914e517-0339-4709-a970-9b32086ab990-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l\" (UID: \"d914e517-0339-4709-a970-9b32086ab990\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l"
Nov 03 22:00:00.390288 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.390169    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlr7w\" (UniqueName: \"kubernetes.io/projected/bbc6ade3-0fe6-45d5-a8a1-36118b848724-kube-api-access-tlr7w\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z\" (UID: \"bbc6ade3-0fe6-45d5-a8a1-36118b848724\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z"
Nov 03 22:00:00.390288 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.390203    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07b51d2-c47b-400a-b786-4d500d736169-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx\" (UID: \"f07b51d2-c47b-400a-b786-4d500d736169\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx"
Nov 03 22:00:00.390288 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.390254    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjx7t\" (UniqueName: \"kubernetes.io/projected/d914e517-0339-4709-a970-9b32086ab990-kube-api-access-pjx7t\") pod \"maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l\" (UID: \"d914e517-0339-4709-a970-9b32086ab990\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l"
Nov 03 22:00:00.390288 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.390283    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53bee6ec-657e-49b0-9e45-2b1c4c1b275f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst\" (UID: \"53bee6ec-657e-49b0-9e45-2b1c4c1b275f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst"
Nov 03 22:00:00.390484 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.390341    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ddmc\" (UniqueName: \"kubernetes.io/projected/f19698dc-e700-4e8e-a07c-7a5f9e74c70f-kube-api-access-7ddmc\") pod \"maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p\" (UID: \"f19698dc-e700-4e8e-a07c-7a5f9e74c70f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p"
Nov 03 22:00:00.390484 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.390401    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1999e701-573c-454b-baca-bf466f9c2ced-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx\" (UID: \"1999e701-573c-454b-baca-bf466f9c2ced\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx"
Nov 03 22:00:00.390484 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.390435    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bdvt\" (UniqueName: \"kubernetes.io/projected/f07b51d2-c47b-400a-b786-4d500d736169-kube-api-access-6bdvt\") pod \"maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx\" (UID: \"f07b51d2-c47b-400a-b786-4d500d736169\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx"
Nov 03 22:00:00.390484 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.390472    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4vd4\" (UniqueName: \"kubernetes.io/projected/1999e701-573c-454b-baca-bf466f9c2ced-kube-api-access-b4vd4\") pod \"maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx\" (UID: \"1999e701-573c-454b-baca-bf466f9c2ced\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx"
Nov 03 22:00:00.390683 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.390503    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc9jd\" (UniqueName: \"kubernetes.io/projected/53bee6ec-657e-49b0-9e45-2b1c4c1b275f-kube-api-access-rc9jd\") pod \"maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst\" (UID: \"53bee6ec-657e-49b0-9e45-2b1c4c1b275f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst"
Nov 03 22:00:00.390683 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.390612    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19698dc-e700-4e8e-a07c-7a5f9e74c70f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p\" (UID: \"f19698dc-e700-4e8e-a07c-7a5f9e74c70f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p"
Nov 03 22:00:00.390796 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.390728    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p"]
Nov 03 22:00:00.391585 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.390974    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc6ade3-0fe6-45d5-a8a1-36118b848724-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z\" (UID: \"bbc6ade3-0fe6-45d5-a8a1-36118b848724\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z"
Nov 03 22:00:00.391585 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.391307    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07b51d2-c47b-400a-b786-4d500d736169-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx\" (UID: \"f07b51d2-c47b-400a-b786-4d500d736169\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx"
Nov 03 22:00:00.391585 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.391340    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53bee6ec-657e-49b0-9e45-2b1c4c1b275f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst\" (UID: \"53bee6ec-657e-49b0-9e45-2b1c4c1b275f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst"
Nov 03 22:00:00.391836 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.391638    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1999e701-573c-454b-baca-bf466f9c2ced-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx\" (UID: \"1999e701-573c-454b-baca-bf466f9c2ced\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx"
Nov 03 22:00:00.478441 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.478319    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bdvt\" (UniqueName: \"kubernetes.io/projected/f07b51d2-c47b-400a-b786-4d500d736169-kube-api-access-6bdvt\") pod \"maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx\" (UID: \"f07b51d2-c47b-400a-b786-4d500d736169\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx"
Nov 03 22:00:00.480412 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.480382    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc9jd\" (UniqueName: \"kubernetes.io/projected/53bee6ec-657e-49b0-9e45-2b1c4c1b275f-kube-api-access-rc9jd\") pod \"maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst\" (UID: \"53bee6ec-657e-49b0-9e45-2b1c4c1b275f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst"
Nov 03 22:00:00.480576 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.480554    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlr7w\" (UniqueName: \"kubernetes.io/projected/bbc6ade3-0fe6-45d5-a8a1-36118b848724-kube-api-access-tlr7w\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z\" (UID: \"bbc6ade3-0fe6-45d5-a8a1-36118b848724\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z"
Nov 03 22:00:00.480702 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.480683    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4vd4\" (UniqueName: \"kubernetes.io/projected/1999e701-573c-454b-baca-bf466f9c2ced-kube-api-access-b4vd4\") pod \"maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx\" (UID: \"1999e701-573c-454b-baca-bf466f9c2ced\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx"
Nov 03 22:00:00.486340 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.486321    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst"
Nov 03 22:00:00.492181 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.492151    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19698dc-e700-4e8e-a07c-7a5f9e74c70f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p\" (UID: \"f19698dc-e700-4e8e-a07c-7a5f9e74c70f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p"
Nov 03 22:00:00.492712 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.492328    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d914e517-0339-4709-a970-9b32086ab990-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l\" (UID: \"d914e517-0339-4709-a970-9b32086ab990\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l"
Nov 03 22:00:00.492712 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.492388    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjx7t\" (UniqueName: \"kubernetes.io/projected/d914e517-0339-4709-a970-9b32086ab990-kube-api-access-pjx7t\") pod \"maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l\" (UID: \"d914e517-0339-4709-a970-9b32086ab990\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l"
Nov 03 22:00:00.492712 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.492454    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ddmc\" (UniqueName: \"kubernetes.io/projected/f19698dc-e700-4e8e-a07c-7a5f9e74c70f-kube-api-access-7ddmc\") pod \"maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p\" (UID: \"f19698dc-e700-4e8e-a07c-7a5f9e74c70f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p"
Nov 03 22:00:00.493327 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.493301    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d914e517-0339-4709-a970-9b32086ab990-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l\" (UID: \"d914e517-0339-4709-a970-9b32086ab990\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l"
Nov 03 22:00:00.493486 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.493457    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx"
Nov 03 22:00:00.494082 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.494063    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19698dc-e700-4e8e-a07c-7a5f9e74c70f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p\" (UID: \"f19698dc-e700-4e8e-a07c-7a5f9e74c70f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p"
Nov 03 22:00:00.508481 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.508432    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx"
Nov 03 22:00:00.537496 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.537460    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z"
Nov 03 22:00:00.552718 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.552648    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ddmc\" (UniqueName: \"kubernetes.io/projected/f19698dc-e700-4e8e-a07c-7a5f9e74c70f-kube-api-access-7ddmc\") pod \"maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p\" (UID: \"f19698dc-e700-4e8e-a07c-7a5f9e74c70f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p"
Nov 03 22:00:00.595256 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.595210    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjx7t\" (UniqueName: \"kubernetes.io/projected/d914e517-0339-4709-a970-9b32086ab990-kube-api-access-pjx7t\") pod \"maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l\" (UID: \"d914e517-0339-4709-a970-9b32086ab990\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l"
Nov 03 22:00:00.841348 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.841272    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l"
Nov 03 22:00:00.848967 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.848938    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p"
Nov 03 22:00:00.884859 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.884832    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst"]
Nov 03 22:00:00.934422 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.934387    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx"]
Nov 03 22:00:00.941010 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.940977    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx"]
Nov 03 22:00:00.960576 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:00:00.960545    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1999e701_573c_454b_baca_bf466f9c2ced.slice/crio-2f9b341d5b51d4d51561cae70e547368b0edb5b53148085035ddb1d0a4c98288 WatchSource:0}: Error finding container 2f9b341d5b51d4d51561cae70e547368b0edb5b53148085035ddb1d0a4c98288: Status 404 returned error can't find the container with id 2f9b341d5b51d4d51561cae70e547368b0edb5b53148085035ddb1d0a4c98288
Nov 03 22:00:00.987247 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:00.987198    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z"]
Nov 03 22:00:00.993200 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:00:00.993174    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc6ade3_0fe6_45d5_a8a1_36118b848724.slice/crio-287a522825f99c52648f71323cbc229156113e50568040e18cb4fc2f6f602f16 WatchSource:0}: Error finding container 287a522825f99c52648f71323cbc229156113e50568040e18cb4fc2f6f602f16: Status 404 returned error can't find the container with id 287a522825f99c52648f71323cbc229156113e50568040e18cb4fc2f6f602f16
Nov 03 22:00:01.006785 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:01.006757    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst" event={"ID":"53bee6ec-657e-49b0-9e45-2b1c4c1b275f","Type":"ContainerStarted","Data":"f7afe9efc894d00baf390fc23d4a9db062b40e8c36e76db8bdd81612de34eeab"}
Nov 03 22:00:01.008085 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:01.008058    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx" event={"ID":"f07b51d2-c47b-400a-b786-4d500d736169","Type":"ContainerStarted","Data":"607c3ae74917683b789003f21e82e544380fcc948522d631f593134a12ff1cc5"}
Nov 03 22:00:01.009537 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:01.009512    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx" event={"ID":"1999e701-573c-454b-baca-bf466f9c2ced","Type":"ContainerStarted","Data":"2f9b341d5b51d4d51561cae70e547368b0edb5b53148085035ddb1d0a4c98288"}
Nov 03 22:00:01.010754 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:01.010733    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z" event={"ID":"bbc6ade3-0fe6-45d5-a8a1-36118b848724","Type":"ContainerStarted","Data":"287a522825f99c52648f71323cbc229156113e50568040e18cb4fc2f6f602f16"}
Nov 03 22:00:01.224173 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:01.224149    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l"]
Nov 03 22:00:01.230374 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:01.230350    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p"]
Nov 03 22:00:01.304642 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:00:01.304564    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd914e517_0339_4709_a970_9b32086ab990.slice/crio-2a9b317e05fd0d65f863cb844ab2584e190deb74a70bc0d4c02634ba4e181c1e WatchSource:0}: Error finding container 2a9b317e05fd0d65f863cb844ab2584e190deb74a70bc0d4c02634ba4e181c1e: Status 404 returned error can't find the container with id 2a9b317e05fd0d65f863cb844ab2584e190deb74a70bc0d4c02634ba4e181c1e
Nov 03 22:00:01.305433 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:00:01.305413    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19698dc_e700_4e8e_a07c_7a5f9e74c70f.slice/crio-c61c618567b569270c0ed178259fe25423dc774eb854685ef204ec2de0be787a WatchSource:0}: Error finding container c61c618567b569270c0ed178259fe25423dc774eb854685ef204ec2de0be787a: Status 404 returned error can't find the container with id c61c618567b569270c0ed178259fe25423dc774eb854685ef204ec2de0be787a
Nov 03 22:00:02.024445 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:02.023221    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst" event={"ID":"53bee6ec-657e-49b0-9e45-2b1c4c1b275f","Type":"ContainerStarted","Data":"1630495b262502fe251f2aba18fb2fe368d11d930de0e0b50834f5e5990d6f29"}
Nov 03 22:00:02.025566 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:02.025538    5053 generic.go:356] "Generic (PLEG): container finished" podID="f07b51d2-c47b-400a-b786-4d500d736169" containerID="11535c4247ee1acc2c9a41e4216933b95cda49c4bb6baa55b10ea26e2eeaa870" exitCode=60
Nov 03 22:00:02.025695 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:02.025640    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx" event={"ID":"f07b51d2-c47b-400a-b786-4d500d736169","Type":"ContainerDied","Data":"11535c4247ee1acc2c9a41e4216933b95cda49c4bb6baa55b10ea26e2eeaa870"}
Nov 03 22:00:02.025882 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:02.025864    5053 scope.go:122] "RemoveContainer" containerID="11535c4247ee1acc2c9a41e4216933b95cda49c4bb6baa55b10ea26e2eeaa870"
Nov 03 22:00:02.028352 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:02.028016    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx" event={"ID":"1999e701-573c-454b-baca-bf466f9c2ced","Type":"ContainerStarted","Data":"908c5551919c28247fb4161e5a75dca6328db3a169075ef5b86264323362db99"}
Nov 03 22:00:02.033819 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:02.033799    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z" event={"ID":"bbc6ade3-0fe6-45d5-a8a1-36118b848724","Type":"ContainerStarted","Data":"3068743fa1141e9d4568986a90cfd4018a4e04d5f994a68799fe7d1b1c3b14ca"}
Nov 03 22:00:02.036571 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:02.036517    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p" event={"ID":"f19698dc-e700-4e8e-a07c-7a5f9e74c70f","Type":"ContainerStarted","Data":"0b54a34e08229c933f3e30b651e44d958226c3d6b8338e4358357959811d2091"}
Nov 03 22:00:02.036571 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:02.036565    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p" event={"ID":"f19698dc-e700-4e8e-a07c-7a5f9e74c70f","Type":"ContainerStarted","Data":"c61c618567b569270c0ed178259fe25423dc774eb854685ef204ec2de0be787a"}
Nov 03 22:00:02.051898 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:02.051855    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l" event={"ID":"d914e517-0339-4709-a970-9b32086ab990","Type":"ContainerStarted","Data":"a47ac81d79eaaf3127b9b5ec1c85dc1b150289e1ea553fdc69246b281a5ce8cc"}
Nov 03 22:00:02.052043 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:02.051916    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l" event={"ID":"d914e517-0339-4709-a970-9b32086ab990","Type":"ContainerStarted","Data":"2a9b317e05fd0d65f863cb844ab2584e190deb74a70bc0d4c02634ba4e181c1e"}
Nov 03 22:00:02.137827 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:02.137777    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst" podStartSLOduration=2.137763516 podStartE2EDuration="2.137763516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:02.134328759 +0000 UTC m=+2716.628250632" watchObservedRunningTime="2026-11-03 22:00:02.137763516 +0000 UTC m=+2716.631685385"
Nov 03 22:00:02.247623 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:02.247563    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx" podStartSLOduration=2.247545385 podStartE2EDuration="2.247545385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:02.246098638 +0000 UTC m=+2716.740020510" watchObservedRunningTime="2026-11-03 22:00:02.247545385 +0000 UTC m=+2716.741467261"
Nov 03 22:00:02.358985 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:02.358932    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p" podStartSLOduration=2.358917808 podStartE2EDuration="2.358917808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:02.358492797 +0000 UTC m=+2716.852414669" watchObservedRunningTime="2026-11-03 22:00:02.358917808 +0000 UTC m=+2716.852839682"
Nov 03 22:00:02.471378 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:02.471315    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z" podStartSLOduration=2.4712968 podStartE2EDuration="2.4712968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:02.470614738 +0000 UTC m=+2716.964536613" watchObservedRunningTime="2026-11-03 22:00:02.4712968 +0000 UTC m=+2716.965218675"
Nov 03 22:00:02.699581 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:02.699525    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l" podStartSLOduration=2.699510218 podStartE2EDuration="2.699510218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:02.695769489 +0000 UTC m=+2717.189691363" watchObservedRunningTime="2026-11-03 22:00:02.699510218 +0000 UTC m=+2717.193432124"
Nov 03 22:00:03.060770 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:03.060724    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx" event={"ID":"f07b51d2-c47b-400a-b786-4d500d736169","Type":"ContainerStarted","Data":"087c8b2db4533a0c3ed6942e97ae7011724d5d81ab547c8831f63457a53488f9"}
Nov 03 22:00:03.062422 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:03.062395    5053 generic.go:356] "Generic (PLEG): container finished" podID="1999e701-573c-454b-baca-bf466f9c2ced" containerID="908c5551919c28247fb4161e5a75dca6328db3a169075ef5b86264323362db99" exitCode=60
Nov 03 22:00:03.062588 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:03.062434    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx" event={"ID":"1999e701-573c-454b-baca-bf466f9c2ced","Type":"ContainerDied","Data":"908c5551919c28247fb4161e5a75dca6328db3a169075ef5b86264323362db99"}
Nov 03 22:00:03.062762 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:03.062744    5053 scope.go:122] "RemoveContainer" containerID="908c5551919c28247fb4161e5a75dca6328db3a169075ef5b86264323362db99"
Nov 03 22:00:03.064257 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:03.064217    5053 generic.go:356] "Generic (PLEG): container finished" podID="bbc6ade3-0fe6-45d5-a8a1-36118b848724" containerID="3068743fa1141e9d4568986a90cfd4018a4e04d5f994a68799fe7d1b1c3b14ca" exitCode=60
Nov 03 22:00:03.064339 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:03.064298    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z" event={"ID":"bbc6ade3-0fe6-45d5-a8a1-36118b848724","Type":"ContainerDied","Data":"3068743fa1141e9d4568986a90cfd4018a4e04d5f994a68799fe7d1b1c3b14ca"}
Nov 03 22:00:03.064545 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:03.064528    5053 scope.go:122] "RemoveContainer" containerID="3068743fa1141e9d4568986a90cfd4018a4e04d5f994a68799fe7d1b1c3b14ca"
Nov 03 22:00:03.066025 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:03.065941    5053 generic.go:356] "Generic (PLEG): container finished" podID="f19698dc-e700-4e8e-a07c-7a5f9e74c70f" containerID="0b54a34e08229c933f3e30b651e44d958226c3d6b8338e4358357959811d2091" exitCode=60
Nov 03 22:00:03.066025 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:03.065964    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p" event={"ID":"f19698dc-e700-4e8e-a07c-7a5f9e74c70f","Type":"ContainerDied","Data":"0b54a34e08229c933f3e30b651e44d958226c3d6b8338e4358357959811d2091"}
Nov 03 22:00:03.066210 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:03.066193    5053 scope.go:122] "RemoveContainer" containerID="0b54a34e08229c933f3e30b651e44d958226c3d6b8338e4358357959811d2091"
Nov 03 22:00:03.067951 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:03.067878    5053 generic.go:356] "Generic (PLEG): container finished" podID="d914e517-0339-4709-a970-9b32086ab990" containerID="a47ac81d79eaaf3127b9b5ec1c85dc1b150289e1ea553fdc69246b281a5ce8cc" exitCode=60
Nov 03 22:00:03.067951 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:03.067932    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l" event={"ID":"d914e517-0339-4709-a970-9b32086ab990","Type":"ContainerDied","Data":"a47ac81d79eaaf3127b9b5ec1c85dc1b150289e1ea553fdc69246b281a5ce8cc"}
Nov 03 22:00:03.068152 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:03.068138    5053 scope.go:122] "RemoveContainer" containerID="a47ac81d79eaaf3127b9b5ec1c85dc1b150289e1ea553fdc69246b281a5ce8cc"
Nov 03 22:00:03.069559 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:03.069540    5053 generic.go:356] "Generic (PLEG): container finished" podID="53bee6ec-657e-49b0-9e45-2b1c4c1b275f" containerID="1630495b262502fe251f2aba18fb2fe368d11d930de0e0b50834f5e5990d6f29" exitCode=60
Nov 03 22:00:03.069663 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:03.069566    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst" event={"ID":"53bee6ec-657e-49b0-9e45-2b1c4c1b275f","Type":"ContainerDied","Data":"1630495b262502fe251f2aba18fb2fe368d11d930de0e0b50834f5e5990d6f29"}
Nov 03 22:00:03.069812 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:03.069796    5053 scope.go:122] "RemoveContainer" containerID="1630495b262502fe251f2aba18fb2fe368d11d930de0e0b50834f5e5990d6f29"
Nov 03 22:00:03.176324 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:03.176262    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx" podStartSLOduration=3.176228639 podStartE2EDuration="3.176228639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:03.172431926 +0000 UTC m=+2717.666353801" watchObservedRunningTime="2026-11-03 22:00:03.176228639 +0000 UTC m=+2717.670150513"
Nov 03 22:00:04.078499 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:04.078461    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z" event={"ID":"bbc6ade3-0fe6-45d5-a8a1-36118b848724","Type":"ContainerStarted","Data":"2838f3d01a144351f7711989d6d390ac6bb7d2c5bf39622711f2fc6050439d25"}
Nov 03 22:00:04.080526 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:04.080501    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p" event={"ID":"f19698dc-e700-4e8e-a07c-7a5f9e74c70f","Type":"ContainerStarted","Data":"24e3fa6d2d13f92eed33ab4625a88a6fb0a701c345ec1d4278762930680e9cff"}
Nov 03 22:00:04.082418 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:04.082394    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l" event={"ID":"d914e517-0339-4709-a970-9b32086ab990","Type":"ContainerStarted","Data":"b0257679cbeebe1ef52a9463559f83cb6774147f6eea43b863964652d548eff6"}
Nov 03 22:00:04.084213 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:04.084189    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst" event={"ID":"53bee6ec-657e-49b0-9e45-2b1c4c1b275f","Type":"ContainerStarted","Data":"3d92a56fea21c7d93080d7fdd24fa78813f7f34e5e80c815e96d00dd7f42676b"}
Nov 03 22:00:04.086049 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:04.086026    5053 generic.go:356] "Generic (PLEG): container finished" podID="f07b51d2-c47b-400a-b786-4d500d736169" containerID="087c8b2db4533a0c3ed6942e97ae7011724d5d81ab547c8831f63457a53488f9" exitCode=60
Nov 03 22:00:04.086173 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:04.086098    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx" event={"ID":"f07b51d2-c47b-400a-b786-4d500d736169","Type":"ContainerDied","Data":"087c8b2db4533a0c3ed6942e97ae7011724d5d81ab547c8831f63457a53488f9"}
Nov 03 22:00:04.086173 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:04.086158    5053 scope.go:122] "RemoveContainer" containerID="11535c4247ee1acc2c9a41e4216933b95cda49c4bb6baa55b10ea26e2eeaa870"
Nov 03 22:00:04.086305 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:04.086286    5053 scope.go:122] "RemoveContainer" containerID="087c8b2db4533a0c3ed6942e97ae7011724d5d81ab547c8831f63457a53488f9"
Nov 03 22:00:04.086518 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:00:04.086491    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx_odh-ai-gateway-infra(f07b51d2-c47b-400a-b786-4d500d736169)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx" podUID="f07b51d2-c47b-400a-b786-4d500d736169"
Nov 03 22:00:04.087927 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:04.087908    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx" event={"ID":"1999e701-573c-454b-baca-bf466f9c2ced","Type":"ContainerStarted","Data":"ac652e8ef4067003605277f16d751f40e8c68bc168f97a9877f7fc324b6eddf6"}
Nov 03 22:00:04.401869 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:04.401821    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l"]
Nov 03 22:00:05.098147 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:05.098117    5053 scope.go:122] "RemoveContainer" containerID="087c8b2db4533a0c3ed6942e97ae7011724d5d81ab547c8831f63457a53488f9"
Nov 03 22:00:05.098668 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:00:05.098507    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx_odh-ai-gateway-infra(f07b51d2-c47b-400a-b786-4d500d736169)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx" podUID="f07b51d2-c47b-400a-b786-4d500d736169"
Nov 03 22:00:05.099952 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:05.099922    5053 generic.go:356] "Generic (PLEG): container finished" podID="1999e701-573c-454b-baca-bf466f9c2ced" containerID="ac652e8ef4067003605277f16d751f40e8c68bc168f97a9877f7fc324b6eddf6" exitCode=60
Nov 03 22:00:05.100101 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:05.099996    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx" event={"ID":"1999e701-573c-454b-baca-bf466f9c2ced","Type":"ContainerDied","Data":"ac652e8ef4067003605277f16d751f40e8c68bc168f97a9877f7fc324b6eddf6"}
Nov 03 22:00:05.100101 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:05.100069    5053 scope.go:122] "RemoveContainer" containerID="908c5551919c28247fb4161e5a75dca6328db3a169075ef5b86264323362db99"
Nov 03 22:00:05.100284 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:05.100264    5053 scope.go:122] "RemoveContainer" containerID="ac652e8ef4067003605277f16d751f40e8c68bc168f97a9877f7fc324b6eddf6"
Nov 03 22:00:05.100593 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:00:05.100573    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx_odh-ai-gateway-infra(1999e701-573c-454b-baca-bf466f9c2ced)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx" podUID="1999e701-573c-454b-baca-bf466f9c2ced"
Nov 03 22:00:05.107796 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:05.107769    5053 generic.go:356] "Generic (PLEG): container finished" podID="bbc6ade3-0fe6-45d5-a8a1-36118b848724" containerID="2838f3d01a144351f7711989d6d390ac6bb7d2c5bf39622711f2fc6050439d25" exitCode=60
Nov 03 22:00:05.107900 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:05.107846    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z" event={"ID":"bbc6ade3-0fe6-45d5-a8a1-36118b848724","Type":"ContainerDied","Data":"2838f3d01a144351f7711989d6d390ac6bb7d2c5bf39622711f2fc6050439d25"}
Nov 03 22:00:05.108282 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:05.108263    5053 scope.go:122] "RemoveContainer" containerID="2838f3d01a144351f7711989d6d390ac6bb7d2c5bf39622711f2fc6050439d25"
Nov 03 22:00:05.108620 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:00:05.108597    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z_odh-ai-gateway-infra(bbc6ade3-0fe6-45d5-a8a1-36118b848724)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z" podUID="bbc6ade3-0fe6-45d5-a8a1-36118b848724"
Nov 03 22:00:05.110208 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:05.110182    5053 generic.go:356] "Generic (PLEG): container finished" podID="f19698dc-e700-4e8e-a07c-7a5f9e74c70f" containerID="24e3fa6d2d13f92eed33ab4625a88a6fb0a701c345ec1d4278762930680e9cff" exitCode=60
Nov 03 22:00:05.110327 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:05.110271    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p" event={"ID":"f19698dc-e700-4e8e-a07c-7a5f9e74c70f","Type":"ContainerDied","Data":"24e3fa6d2d13f92eed33ab4625a88a6fb0a701c345ec1d4278762930680e9cff"}
Nov 03 22:00:05.110536 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:05.110520    5053 scope.go:122] "RemoveContainer" containerID="24e3fa6d2d13f92eed33ab4625a88a6fb0a701c345ec1d4278762930680e9cff"
Nov 03 22:00:05.110842 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:00:05.110816    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p_odh-ai-gateway-infra(f19698dc-e700-4e8e-a07c-7a5f9e74c70f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p" podUID="f19698dc-e700-4e8e-a07c-7a5f9e74c70f"
Nov 03 22:00:05.112555 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:05.112531    5053 generic.go:356] "Generic (PLEG): container finished" podID="d914e517-0339-4709-a970-9b32086ab990" containerID="b0257679cbeebe1ef52a9463559f83cb6774147f6eea43b863964652d548eff6" exitCode=60
Nov 03 22:00:05.112673 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:05.112657    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l" event={"ID":"d914e517-0339-4709-a970-9b32086ab990","Type":"ContainerDied","Data":"b0257679cbeebe1ef52a9463559f83cb6774147f6eea43b863964652d548eff6"}
Nov 03 22:00:05.112903 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:05.112888    5053 scope.go:122] "RemoveContainer" containerID="b0257679cbeebe1ef52a9463559f83cb6774147f6eea43b863964652d548eff6"
Nov 03 22:00:05.113172 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:00:05.113152    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l_odh-ai-gateway-infra(d914e517-0339-4709-a970-9b32086ab990)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l" podUID="d914e517-0339-4709-a970-9b32086ab990"
Nov 03 22:00:05.115134 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:05.115111    5053 generic.go:356] "Generic (PLEG): container finished" podID="53bee6ec-657e-49b0-9e45-2b1c4c1b275f" containerID="3d92a56fea21c7d93080d7fdd24fa78813f7f34e5e80c815e96d00dd7f42676b" exitCode=60
Nov 03 22:00:05.115269 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:05.115174    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst" event={"ID":"53bee6ec-657e-49b0-9e45-2b1c4c1b275f","Type":"ContainerDied","Data":"3d92a56fea21c7d93080d7fdd24fa78813f7f34e5e80c815e96d00dd7f42676b"}
Nov 03 22:00:05.115465 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:05.115450    5053 scope.go:122] "RemoveContainer" containerID="3d92a56fea21c7d93080d7fdd24fa78813f7f34e5e80c815e96d00dd7f42676b"
Nov 03 22:00:05.115691 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:00:05.115671    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst_odh-ai-gateway-infra(53bee6ec-657e-49b0-9e45-2b1c4c1b275f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst" podUID="53bee6ec-657e-49b0-9e45-2b1c4c1b275f"
Nov 03 22:00:05.117477 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:05.117460    5053 scope.go:122] "RemoveContainer" containerID="3068743fa1141e9d4568986a90cfd4018a4e04d5f994a68799fe7d1b1c3b14ca"
Nov 03 22:00:05.132937 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:05.132915    5053 scope.go:122] "RemoveContainer" containerID="0b54a34e08229c933f3e30b651e44d958226c3d6b8338e4358357959811d2091"
Nov 03 22:00:05.149144 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:05.149123    5053 scope.go:122] "RemoveContainer" containerID="a47ac81d79eaaf3127b9b5ec1c85dc1b150289e1ea553fdc69246b281a5ce8cc"
Nov 03 22:00:05.173270 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:05.169682    5053 scope.go:122] "RemoveContainer" containerID="1630495b262502fe251f2aba18fb2fe368d11d930de0e0b50834f5e5990d6f29"
Nov 03 22:00:06.131022 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:06.130992    5053 scope.go:122] "RemoveContainer" containerID="ac652e8ef4067003605277f16d751f40e8c68bc168f97a9877f7fc324b6eddf6"
Nov 03 22:00:06.131538 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:00:06.131372    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx_odh-ai-gateway-infra(1999e701-573c-454b-baca-bf466f9c2ced)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx" podUID="1999e701-573c-454b-baca-bf466f9c2ced"
Nov 03 22:00:06.132881 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:06.132853    5053 scope.go:122] "RemoveContainer" containerID="2838f3d01a144351f7711989d6d390ac6bb7d2c5bf39622711f2fc6050439d25"
Nov 03 22:00:06.133177 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:00:06.133148    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z_odh-ai-gateway-infra(bbc6ade3-0fe6-45d5-a8a1-36118b848724)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z" podUID="bbc6ade3-0fe6-45d5-a8a1-36118b848724"
Nov 03 22:00:06.134467 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:06.134446    5053 scope.go:122] "RemoveContainer" containerID="24e3fa6d2d13f92eed33ab4625a88a6fb0a701c345ec1d4278762930680e9cff"
Nov 03 22:00:06.134710 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:00:06.134691    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p_odh-ai-gateway-infra(f19698dc-e700-4e8e-a07c-7a5f9e74c70f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p" podUID="f19698dc-e700-4e8e-a07c-7a5f9e74c70f"
Nov 03 22:00:06.138644 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:06.138618    5053 scope.go:122] "RemoveContainer" containerID="3d92a56fea21c7d93080d7fdd24fa78813f7f34e5e80c815e96d00dd7f42676b"
Nov 03 22:00:06.139016 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:00:06.138992    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst_odh-ai-gateway-infra(53bee6ec-657e-49b0-9e45-2b1c4c1b275f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst" podUID="53bee6ec-657e-49b0-9e45-2b1c4c1b275f"
Nov 03 22:00:06.321805 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:06.321774    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l"
Nov 03 22:00:06.415546 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:06.415453    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d914e517-0339-4709-a970-9b32086ab990-kube-api-access-pjx7t\" (UniqueName: \"kubernetes.io/projected/d914e517-0339-4709-a970-9b32086ab990-kube-api-access-pjx7t\") pod \"d914e517-0339-4709-a970-9b32086ab990\" (UID: \"d914e517-0339-4709-a970-9b32086ab990\") "
Nov 03 22:00:06.415546 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:06.415502    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d914e517-0339-4709-a970-9b32086ab990-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d914e517-0339-4709-a970-9b32086ab990-maas-api-service-ca\") pod \"d914e517-0339-4709-a970-9b32086ab990\" (UID: \"d914e517-0339-4709-a970-9b32086ab990\") "
Nov 03 22:00:06.415965 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:06.415940    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d914e517-0339-4709-a970-9b32086ab990-maas-api-service-ca" pod "d914e517-0339-4709-a970-9b32086ab990" (UID: "d914e517-0339-4709-a970-9b32086ab990"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:06.417971 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:06.417944    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d914e517-0339-4709-a970-9b32086ab990-kube-api-access-pjx7t" pod "d914e517-0339-4709-a970-9b32086ab990" (UID: "d914e517-0339-4709-a970-9b32086ab990"). InnerVolumeSpecName "kube-api-access-pjx7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:06.472149 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:06.472107    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl"]
Nov 03 22:00:06.475441 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:06.475417    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd2a9f-phbrl"]
Nov 03 22:00:06.517135 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:06.517057    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjx7t\" (UniqueName: \"kubernetes.io/projected/d914e517-0339-4709-a970-9b32086ab990-kube-api-access-pjx7t\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:06.517135 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:06.517097    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d914e517-0339-4709-a970-9b32086ab990-maas-api-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:07.147773 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:07.147739    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l"
Nov 03 22:00:07.147773 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:07.147757    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l" event={"ID":"d914e517-0339-4709-a970-9b32086ab990","Type":"ContainerDied","Data":"2a9b317e05fd0d65f863cb844ab2584e190deb74a70bc0d4c02634ba4e181c1e"}
Nov 03 22:00:07.148358 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:07.147822    5053 scope.go:122] "RemoveContainer" containerID="b0257679cbeebe1ef52a9463559f83cb6774147f6eea43b863964652d548eff6"
Nov 03 22:00:07.312288 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:07.312249    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l"]
Nov 03 22:00:07.320620 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:07.320587    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cd2a9f-29895720-dcx5l"]
Nov 03 22:00:07.669609 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:07.669577    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d914e517-0339-4709-a970-9b32086ab990" path="/var/lib/kubelet/pods/d914e517-0339-4709-a970-9b32086ab990/volumes"
Nov 03 22:00:07.670017 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:07.670002    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7212b57-eddc-4cb8-8657-f3842ab5ad5d" path="/var/lib/kubelet/pods/f7212b57-eddc-4cb8-8657-f3842ab5ad5d/volumes"
Nov 03 22:00:10.928263 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:00:10.928203    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:00:16.659606 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:16.659571    5053 scope.go:122] "RemoveContainer" containerID="2838f3d01a144351f7711989d6d390ac6bb7d2c5bf39622711f2fc6050439d25"
Nov 03 22:00:16.659996 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:16.659697    5053 scope.go:122] "RemoveContainer" containerID="087c8b2db4533a0c3ed6942e97ae7011724d5d81ab547c8831f63457a53488f9"
Nov 03 22:00:16.659996 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:16.659831    5053 scope.go:122] "RemoveContainer" containerID="ac652e8ef4067003605277f16d751f40e8c68bc168f97a9877f7fc324b6eddf6"
Nov 03 22:00:17.279280 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:17.279228    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx" event={"ID":"f07b51d2-c47b-400a-b786-4d500d736169","Type":"ContainerStarted","Data":"20d1387dbccfc997ecee43e016da003869d281ae6641b5f486d71497c9a4bcbb"}
Nov 03 22:00:17.281136 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:17.281108    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx" event={"ID":"1999e701-573c-454b-baca-bf466f9c2ced","Type":"ContainerStarted","Data":"d36ca9ae77c01aeeb7ea0904c304f007b767ae224aec1892029fad898c38b246"}
Nov 03 22:00:17.282980 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:17.282960    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z" event={"ID":"bbc6ade3-0fe6-45d5-a8a1-36118b848724","Type":"ContainerStarted","Data":"548043157fe9c099b513645d371c6ad3b79b6170c76e7c260d05fb662057c28e"}
Nov 03 22:00:17.660376 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:17.660354    5053 scope.go:122] "RemoveContainer" containerID="3d92a56fea21c7d93080d7fdd24fa78813f7f34e5e80c815e96d00dd7f42676b"
Nov 03 22:00:18.294020 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:18.293987    5053 generic.go:356] "Generic (PLEG): container finished" podID="bbc6ade3-0fe6-45d5-a8a1-36118b848724" containerID="548043157fe9c099b513645d371c6ad3b79b6170c76e7c260d05fb662057c28e" exitCode=60
Nov 03 22:00:18.294202 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:18.294073    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z" event={"ID":"bbc6ade3-0fe6-45d5-a8a1-36118b848724","Type":"ContainerDied","Data":"548043157fe9c099b513645d371c6ad3b79b6170c76e7c260d05fb662057c28e"}
Nov 03 22:00:18.294202 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:18.294144    5053 scope.go:122] "RemoveContainer" containerID="2838f3d01a144351f7711989d6d390ac6bb7d2c5bf39622711f2fc6050439d25"
Nov 03 22:00:18.294544 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:18.294527    5053 scope.go:122] "RemoveContainer" containerID="548043157fe9c099b513645d371c6ad3b79b6170c76e7c260d05fb662057c28e"
Nov 03 22:00:18.294835 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:00:18.294816    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z_odh-ai-gateway-infra(bbc6ade3-0fe6-45d5-a8a1-36118b848724)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z" podUID="bbc6ade3-0fe6-45d5-a8a1-36118b848724"
Nov 03 22:00:18.297317 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:18.297285    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst" event={"ID":"53bee6ec-657e-49b0-9e45-2b1c4c1b275f","Type":"ContainerStarted","Data":"23c480b2909be89b377d4c80d1878ef4495e4282b2e45fb79479e3cef19f9641"}
Nov 03 22:00:18.299691 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:18.299665    5053 generic.go:356] "Generic (PLEG): container finished" podID="f07b51d2-c47b-400a-b786-4d500d736169" containerID="20d1387dbccfc997ecee43e016da003869d281ae6641b5f486d71497c9a4bcbb" exitCode=60
Nov 03 22:00:18.299798 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:18.299748    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx" event={"ID":"f07b51d2-c47b-400a-b786-4d500d736169","Type":"ContainerDied","Data":"20d1387dbccfc997ecee43e016da003869d281ae6641b5f486d71497c9a4bcbb"}
Nov 03 22:00:18.300131 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:18.300111    5053 scope.go:122] "RemoveContainer" containerID="20d1387dbccfc997ecee43e016da003869d281ae6641b5f486d71497c9a4bcbb"
Nov 03 22:00:18.300462 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:00:18.300414    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx_odh-ai-gateway-infra(f07b51d2-c47b-400a-b786-4d500d736169)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx" podUID="f07b51d2-c47b-400a-b786-4d500d736169"
Nov 03 22:00:18.302073 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:18.302039    5053 generic.go:356] "Generic (PLEG): container finished" podID="1999e701-573c-454b-baca-bf466f9c2ced" containerID="d36ca9ae77c01aeeb7ea0904c304f007b767ae224aec1892029fad898c38b246" exitCode=60
Nov 03 22:00:18.302184 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:18.302114    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx" event={"ID":"1999e701-573c-454b-baca-bf466f9c2ced","Type":"ContainerDied","Data":"d36ca9ae77c01aeeb7ea0904c304f007b767ae224aec1892029fad898c38b246"}
Nov 03 22:00:18.302483 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:18.302464    5053 scope.go:122] "RemoveContainer" containerID="d36ca9ae77c01aeeb7ea0904c304f007b767ae224aec1892029fad898c38b246"
Nov 03 22:00:18.302760 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:00:18.302740    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx_odh-ai-gateway-infra(1999e701-573c-454b-baca-bf466f9c2ced)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx" podUID="1999e701-573c-454b-baca-bf466f9c2ced"
Nov 03 22:00:18.311120 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:18.311096    5053 scope.go:122] "RemoveContainer" containerID="087c8b2db4533a0c3ed6942e97ae7011724d5d81ab547c8831f63457a53488f9"
Nov 03 22:00:18.324188 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:18.324169    5053 scope.go:122] "RemoveContainer" containerID="ac652e8ef4067003605277f16d751f40e8c68bc168f97a9877f7fc324b6eddf6"
Nov 03 22:00:18.404506 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:18.404473    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx"]
Nov 03 22:00:18.516363 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:18.516333    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z"]
Nov 03 22:00:18.628318 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:18.628289    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx"]
Nov 03 22:00:18.660009 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:18.659983    5053 scope.go:122] "RemoveContainer" containerID="24e3fa6d2d13f92eed33ab4625a88a6fb0a701c345ec1d4278762930680e9cff"
Nov 03 22:00:19.318024 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.317953    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p" event={"ID":"f19698dc-e700-4e8e-a07c-7a5f9e74c70f","Type":"ContainerStarted","Data":"cc12824bf999dee7be27bf28db21fc4ea8b73124f21b2aedf1fa5d30619accc0"}
Nov 03 22:00:19.321166 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.321135    5053 generic.go:356] "Generic (PLEG): container finished" podID="53bee6ec-657e-49b0-9e45-2b1c4c1b275f" containerID="23c480b2909be89b377d4c80d1878ef4495e4282b2e45fb79479e3cef19f9641" exitCode=60
Nov 03 22:00:19.321318 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.321196    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst" event={"ID":"53bee6ec-657e-49b0-9e45-2b1c4c1b275f","Type":"ContainerDied","Data":"23c480b2909be89b377d4c80d1878ef4495e4282b2e45fb79479e3cef19f9641"}
Nov 03 22:00:19.321318 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.321288    5053 scope.go:122] "RemoveContainer" containerID="3d92a56fea21c7d93080d7fdd24fa78813f7f34e5e80c815e96d00dd7f42676b"
Nov 03 22:00:19.321748 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.321715    5053 scope.go:122] "RemoveContainer" containerID="23c480b2909be89b377d4c80d1878ef4495e4282b2e45fb79479e3cef19f9641"
Nov 03 22:00:19.322097 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:00:19.322068    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst_odh-ai-gateway-infra(53bee6ec-657e-49b0-9e45-2b1c4c1b275f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst" podUID="53bee6ec-657e-49b0-9e45-2b1c4c1b275f"
Nov 03 22:00:19.510176 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.509580    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx"
Nov 03 22:00:19.593453 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.593426    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx"
Nov 03 22:00:19.600520 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.600366    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z"
Nov 03 22:00:19.611966 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.611937    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f07b51d2-c47b-400a-b786-4d500d736169-kube-api-access-6bdvt\" (UniqueName: \"kubernetes.io/projected/f07b51d2-c47b-400a-b786-4d500d736169-kube-api-access-6bdvt\") pod \"f07b51d2-c47b-400a-b786-4d500d736169\" (UID: \"f07b51d2-c47b-400a-b786-4d500d736169\") "
Nov 03 22:00:19.612121 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.612018    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f07b51d2-c47b-400a-b786-4d500d736169-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07b51d2-c47b-400a-b786-4d500d736169-maas-api-service-ca\") pod \"f07b51d2-c47b-400a-b786-4d500d736169\" (UID: \"f07b51d2-c47b-400a-b786-4d500d736169\") "
Nov 03 22:00:19.612121 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.612104    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1999e701-573c-454b-baca-bf466f9c2ced-kube-api-access-b4vd4\" (UniqueName: \"kubernetes.io/projected/1999e701-573c-454b-baca-bf466f9c2ced-kube-api-access-b4vd4\") pod \"1999e701-573c-454b-baca-bf466f9c2ced\" (UID: \"1999e701-573c-454b-baca-bf466f9c2ced\") "
Nov 03 22:00:19.612228 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.612141    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1999e701-573c-454b-baca-bf466f9c2ced-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1999e701-573c-454b-baca-bf466f9c2ced-maas-api-service-ca\") pod \"1999e701-573c-454b-baca-bf466f9c2ced\" (UID: \"1999e701-573c-454b-baca-bf466f9c2ced\") "
Nov 03 22:00:19.612832 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.612809    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1999e701-573c-454b-baca-bf466f9c2ced-maas-api-service-ca" pod "1999e701-573c-454b-baca-bf466f9c2ced" (UID: "1999e701-573c-454b-baca-bf466f9c2ced"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:19.613295 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.613037    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f07b51d2-c47b-400a-b786-4d500d736169-maas-api-service-ca" pod "f07b51d2-c47b-400a-b786-4d500d736169" (UID: "f07b51d2-c47b-400a-b786-4d500d736169"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:19.615080 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.615047    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1999e701-573c-454b-baca-bf466f9c2ced-kube-api-access-b4vd4" pod "1999e701-573c-454b-baca-bf466f9c2ced" (UID: "1999e701-573c-454b-baca-bf466f9c2ced"). InnerVolumeSpecName "kube-api-access-b4vd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:19.615641 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.615620    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07b51d2-c47b-400a-b786-4d500d736169-kube-api-access-6bdvt" pod "f07b51d2-c47b-400a-b786-4d500d736169" (UID: "f07b51d2-c47b-400a-b786-4d500d736169"). InnerVolumeSpecName "kube-api-access-6bdvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:19.654068 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.654034    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst"]
Nov 03 22:00:19.713644 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.713614    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc6ade3-0fe6-45d5-a8a1-36118b848724-kube-api-access-tlr7w\" (UniqueName: \"kubernetes.io/projected/bbc6ade3-0fe6-45d5-a8a1-36118b848724-kube-api-access-tlr7w\") pod \"bbc6ade3-0fe6-45d5-a8a1-36118b848724\" (UID: \"bbc6ade3-0fe6-45d5-a8a1-36118b848724\") "
Nov 03 22:00:19.713811 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.713726    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbc6ade3-0fe6-45d5-a8a1-36118b848724-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc6ade3-0fe6-45d5-a8a1-36118b848724-maas-api-service-ca\") pod \"bbc6ade3-0fe6-45d5-a8a1-36118b848724\" (UID: \"bbc6ade3-0fe6-45d5-a8a1-36118b848724\") "
Nov 03 22:00:19.714286 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.714194    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4vd4\" (UniqueName: \"kubernetes.io/projected/1999e701-573c-454b-baca-bf466f9c2ced-kube-api-access-b4vd4\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:19.714286 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.714225    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1999e701-573c-454b-baca-bf466f9c2ced-maas-api-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:19.714286 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.714193    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc6ade3-0fe6-45d5-a8a1-36118b848724-maas-api-service-ca" pod "bbc6ade3-0fe6-45d5-a8a1-36118b848724" (UID: "bbc6ade3-0fe6-45d5-a8a1-36118b848724"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:19.714564 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.714545    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bdvt\" (UniqueName: \"kubernetes.io/projected/f07b51d2-c47b-400a-b786-4d500d736169-kube-api-access-6bdvt\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:19.714635 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.714570    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07b51d2-c47b-400a-b786-4d500d736169-maas-api-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:19.716334 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.716311    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc6ade3-0fe6-45d5-a8a1-36118b848724-kube-api-access-tlr7w" pod "bbc6ade3-0fe6-45d5-a8a1-36118b848724" (UID: "bbc6ade3-0fe6-45d5-a8a1-36118b848724"). InnerVolumeSpecName "kube-api-access-tlr7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:19.816400 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.816309    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlr7w\" (UniqueName: \"kubernetes.io/projected/bbc6ade3-0fe6-45d5-a8a1-36118b848724-kube-api-access-tlr7w\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:19.816400 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:19.816335    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc6ade3-0fe6-45d5-a8a1-36118b848724-maas-api-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:20.132783 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.132702    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3112e9fa-68d69"]
Nov 03 22:00:20.133387 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.133371    5053 cpu_manager.go:414] "containerMap: removing container" podUID="bbc6ade3-0fe6-45d5-a8a1-36118b848724" containerName="cleanup"
Nov 03 22:00:20.133430 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.133387    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc6ade3-0fe6-45d5-a8a1-36118b848724" containerName="cleanup"
Nov 03 22:00:20.133430 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.133399    5053 cpu_manager.go:414] "containerMap: removing container" podUID="1999e701-573c-454b-baca-bf466f9c2ced" containerName="cleanup"
Nov 03 22:00:20.133430 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.133404    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1999e701-573c-454b-baca-bf466f9c2ced" containerName="cleanup"
Nov 03 22:00:20.133430 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.133414    5053 cpu_manager.go:414] "containerMap: removing container" podUID="f07b51d2-c47b-400a-b786-4d500d736169" containerName="cleanup"
Nov 03 22:00:20.133430 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.133418    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07b51d2-c47b-400a-b786-4d500d736169" containerName="cleanup"
Nov 03 22:00:20.133581 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.133446    5053 cpu_manager.go:414] "containerMap: removing container" podUID="d914e517-0339-4709-a970-9b32086ab990" containerName="cleanup"
Nov 03 22:00:20.133581 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.133451    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d914e517-0339-4709-a970-9b32086ab990" containerName="cleanup"
Nov 03 22:00:20.133581 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.133458    5053 cpu_manager.go:414] "containerMap: removing container" podUID="bbc6ade3-0fe6-45d5-a8a1-36118b848724" containerName="cleanup"
Nov 03 22:00:20.133581 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.133462    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc6ade3-0fe6-45d5-a8a1-36118b848724" containerName="cleanup"
Nov 03 22:00:20.133581 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.133473    5053 cpu_manager.go:414] "containerMap: removing container" podUID="1999e701-573c-454b-baca-bf466f9c2ced" containerName="cleanup"
Nov 03 22:00:20.133581 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.133477    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1999e701-573c-454b-baca-bf466f9c2ced" containerName="cleanup"
Nov 03 22:00:20.133581 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.133486    5053 cpu_manager.go:414] "containerMap: removing container" podUID="d914e517-0339-4709-a970-9b32086ab990" containerName="cleanup"
Nov 03 22:00:20.133581 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.133490    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d914e517-0339-4709-a970-9b32086ab990" containerName="cleanup"
Nov 03 22:00:20.133581 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.133503    5053 cpu_manager.go:414] "containerMap: removing container" podUID="f07b51d2-c47b-400a-b786-4d500d736169" containerName="cleanup"
Nov 03 22:00:20.133581 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.133507    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07b51d2-c47b-400a-b786-4d500d736169" containerName="cleanup"
Nov 03 22:00:20.133859 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.133588    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc6ade3-0fe6-45d5-a8a1-36118b848724" containerName="cleanup"
Nov 03 22:00:20.133859 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.133595    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="1999e701-573c-454b-baca-bf466f9c2ced" containerName="cleanup"
Nov 03 22:00:20.133859 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.133601    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="f07b51d2-c47b-400a-b786-4d500d736169" containerName="cleanup"
Nov 03 22:00:20.133859 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.133607    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc6ade3-0fe6-45d5-a8a1-36118b848724" containerName="cleanup"
Nov 03 22:00:20.133859 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.133618    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="d914e517-0339-4709-a970-9b32086ab990" containerName="cleanup"
Nov 03 22:00:20.133859 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.133624    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="1999e701-573c-454b-baca-bf466f9c2ced" containerName="cleanup"
Nov 03 22:00:20.133859 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.133629    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="d914e517-0339-4709-a970-9b32086ab990" containerName="cleanup"
Nov 03 22:00:20.133859 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.133636    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="f07b51d2-c47b-400a-b786-4d500d736169" containerName="cleanup"
Nov 03 22:00:20.137072 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.137054    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3112e9fa-68d69"
Nov 03 22:00:20.190948 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.190915    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:00:20.192900 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.192871    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3112e9fa-68d69"]
Nov 03 22:00:20.221011 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.220972    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7e82a8c-058a-467f-870d-361ef4b86532-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3112e9fa-68d69\" (UID: \"a7e82a8c-058a-467f-870d-361ef4b86532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3112e9fa-68d69"
Nov 03 22:00:20.221218 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.221031    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg2tp\" (UniqueName: \"kubernetes.io/projected/a7e82a8c-058a-467f-870d-361ef4b86532-kube-api-access-hg2tp\") pod \"maas-api-revoke-sub-3112e9fa-68d69\" (UID: \"a7e82a8c-058a-467f-870d-361ef4b86532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3112e9fa-68d69"
Nov 03 22:00:20.221218 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.221105    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7e82a8c-058a-467f-870d-361ef4b86532-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3112e9fa-68d69\" (UID: \"a7e82a8c-058a-467f-870d-361ef4b86532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3112e9fa-68d69"
Nov 03 22:00:20.322009 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.321976    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7e82a8c-058a-467f-870d-361ef4b86532-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3112e9fa-68d69\" (UID: \"a7e82a8c-058a-467f-870d-361ef4b86532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3112e9fa-68d69"
Nov 03 22:00:20.322531 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.322027    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg2tp\" (UniqueName: \"kubernetes.io/projected/a7e82a8c-058a-467f-870d-361ef4b86532-kube-api-access-hg2tp\") pod \"maas-api-revoke-sub-3112e9fa-68d69\" (UID: \"a7e82a8c-058a-467f-870d-361ef4b86532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3112e9fa-68d69"
Nov 03 22:00:20.322531 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.322061    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7e82a8c-058a-467f-870d-361ef4b86532-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3112e9fa-68d69\" (UID: \"a7e82a8c-058a-467f-870d-361ef4b86532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3112e9fa-68d69"
Nov 03 22:00:20.322890 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.322866    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7e82a8c-058a-467f-870d-361ef4b86532-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3112e9fa-68d69\" (UID: \"a7e82a8c-058a-467f-870d-361ef4b86532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3112e9fa-68d69"
Nov 03 22:00:20.324617 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.324591    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7e82a8c-058a-467f-870d-361ef4b86532-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3112e9fa-68d69\" (UID: \"a7e82a8c-058a-467f-870d-361ef4b86532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3112e9fa-68d69"
Nov 03 22:00:20.330622 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.330600    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z"
Nov 03 22:00:20.330765 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.330597    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z" event={"ID":"bbc6ade3-0fe6-45d5-a8a1-36118b848724","Type":"ContainerDied","Data":"287a522825f99c52648f71323cbc229156113e50568040e18cb4fc2f6f602f16"}
Nov 03 22:00:20.330765 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.330736    5053 scope.go:122] "RemoveContainer" containerID="548043157fe9c099b513645d371c6ad3b79b6170c76e7c260d05fb662057c28e"
Nov 03 22:00:20.332773 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.332742    5053 generic.go:356] "Generic (PLEG): container finished" podID="f19698dc-e700-4e8e-a07c-7a5f9e74c70f" containerID="cc12824bf999dee7be27bf28db21fc4ea8b73124f21b2aedf1fa5d30619accc0" exitCode=60
Nov 03 22:00:20.332902 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.332880    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p" event={"ID":"f19698dc-e700-4e8e-a07c-7a5f9e74c70f","Type":"ContainerDied","Data":"cc12824bf999dee7be27bf28db21fc4ea8b73124f21b2aedf1fa5d30619accc0"}
Nov 03 22:00:20.333457 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.333308    5053 scope.go:122] "RemoveContainer" containerID="cc12824bf999dee7be27bf28db21fc4ea8b73124f21b2aedf1fa5d30619accc0"
Nov 03 22:00:20.333628 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:00:20.333608    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p_odh-ai-gateway-infra(f19698dc-e700-4e8e-a07c-7a5f9e74c70f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p" podUID="f19698dc-e700-4e8e-a07c-7a5f9e74c70f"
Nov 03 22:00:20.336784 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.336760    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx" event={"ID":"f07b51d2-c47b-400a-b786-4d500d736169","Type":"ContainerDied","Data":"607c3ae74917683b789003f21e82e544380fcc948522d631f593134a12ff1cc5"}
Nov 03 22:00:20.336904 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.336888    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx"
Nov 03 22:00:20.338215 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.338186    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx" event={"ID":"1999e701-573c-454b-baca-bf466f9c2ced","Type":"ContainerDied","Data":"2f9b341d5b51d4d51561cae70e547368b0edb5b53148085035ddb1d0a4c98288"}
Nov 03 22:00:20.338344 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.338316    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx"
Nov 03 22:00:20.343040 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.343025    5053 scope.go:122] "RemoveContainer" containerID="24e3fa6d2d13f92eed33ab4625a88a6fb0a701c345ec1d4278762930680e9cff"
Nov 03 22:00:20.357556 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.357535    5053 scope.go:122] "RemoveContainer" containerID="20d1387dbccfc997ecee43e016da003869d281ae6641b5f486d71497c9a4bcbb"
Nov 03 22:00:20.368646 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.368617    5053 scope.go:122] "RemoveContainer" containerID="d36ca9ae77c01aeeb7ea0904c304f007b767ae224aec1892029fad898c38b246"
Nov 03 22:00:20.381267 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.381226    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg2tp\" (UniqueName: \"kubernetes.io/projected/a7e82a8c-058a-467f-870d-361ef4b86532-kube-api-access-hg2tp\") pod \"maas-api-revoke-sub-3112e9fa-68d69\" (UID: \"a7e82a8c-058a-467f-870d-361ef4b86532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3112e9fa-68d69"
Nov 03 22:00:20.455691 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.455661    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3112e9fa-68d69"
Nov 03 22:00:20.514276 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.513077    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx"]
Nov 03 22:00:20.519248 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.518649    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895720-dhgwx"]
Nov 03 22:00:20.555131 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.555092    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p"]
Nov 03 22:00:20.632700 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.626219    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst"
Nov 03 22:00:20.729612 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.729531    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z"]
Nov 03 22:00:20.731845 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.731817    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53bee6ec-657e-49b0-9e45-2b1c4c1b275f-kube-api-access-rc9jd\" (UniqueName: \"kubernetes.io/projected/53bee6ec-657e-49b0-9e45-2b1c4c1b275f-kube-api-access-rc9jd\") pod \"53bee6ec-657e-49b0-9e45-2b1c4c1b275f\" (UID: \"53bee6ec-657e-49b0-9e45-2b1c4c1b275f\") "
Nov 03 22:00:20.731968 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.731950    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53bee6ec-657e-49b0-9e45-2b1c4c1b275f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53bee6ec-657e-49b0-9e45-2b1c4c1b275f-maas-api-service-ca\") pod \"53bee6ec-657e-49b0-9e45-2b1c4c1b275f\" (UID: \"53bee6ec-657e-49b0-9e45-2b1c4c1b275f\") "
Nov 03 22:00:20.732397 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.732371    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53bee6ec-657e-49b0-9e45-2b1c4c1b275f-maas-api-service-ca" pod "53bee6ec-657e-49b0-9e45-2b1c4c1b275f" (UID: "53bee6ec-657e-49b0-9e45-2b1c4c1b275f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:20.732513 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.732494    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53bee6ec-657e-49b0-9e45-2b1c4c1b275f-maas-api-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:20.732961 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.732941    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895720-59g9z"]
Nov 03 22:00:20.734191 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.734169    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53bee6ec-657e-49b0-9e45-2b1c4c1b275f-kube-api-access-rc9jd" pod "53bee6ec-657e-49b0-9e45-2b1c4c1b275f" (UID: "53bee6ec-657e-49b0-9e45-2b1c4c1b275f"). InnerVolumeSpecName "kube-api-access-rc9jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:20.833854 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.833822    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc9jd\" (UniqueName: \"kubernetes.io/projected/53bee6ec-657e-49b0-9e45-2b1c4c1b275f-kube-api-access-rc9jd\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:20.842967 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.842922    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx"]
Nov 03 22:00:20.848015 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.847985    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4644ae-29895720-mclqx"]
Nov 03 22:00:20.890332 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:20.890304    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3112e9fa-68d69"]
Nov 03 22:00:20.891366 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:00:20.891345    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e82a8c_058a_467f_870d_361ef4b86532.slice/crio-cc72af4093c35088d6e5bd6486efb0a4906cbae293f40957db228ba831a1a45b WatchSource:0}: Error finding container cc72af4093c35088d6e5bd6486efb0a4906cbae293f40957db228ba831a1a45b: Status 404 returned error can't find the container with id cc72af4093c35088d6e5bd6486efb0a4906cbae293f40957db228ba831a1a45b
Nov 03 22:00:21.347023 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:21.346987    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3112e9fa-68d69" event={"ID":"a7e82a8c-058a-467f-870d-361ef4b86532","Type":"ContainerStarted","Data":"0d1f620d4e229a01fc761c02d028d1206ef841079ad91b74b652df8283e61bff"}
Nov 03 22:00:21.347463 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:21.347041    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3112e9fa-68d69" event={"ID":"a7e82a8c-058a-467f-870d-361ef4b86532","Type":"ContainerStarted","Data":"cc72af4093c35088d6e5bd6486efb0a4906cbae293f40957db228ba831a1a45b"}
Nov 03 22:00:21.352529 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:21.352506    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst" event={"ID":"53bee6ec-657e-49b0-9e45-2b1c4c1b275f","Type":"ContainerDied","Data":"f7afe9efc894d00baf390fc23d4a9db062b40e8c36e76db8bdd81612de34eeab"}
Nov 03 22:00:21.352727 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:21.352529    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst"
Nov 03 22:00:21.352727 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:21.352550    5053 scope.go:122] "RemoveContainer" containerID="23c480b2909be89b377d4c80d1878ef4495e4282b2e45fb79479e3cef19f9641"
Nov 03 22:00:21.463631 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:21.463562    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3112e9fa-68d69" podStartSLOduration=1.463543259 podStartE2EDuration="1.463543259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:21.460300814 +0000 UTC m=+2735.954222693" watchObservedRunningTime="2026-11-03 22:00:21.463543259 +0000 UTC m=+2735.957465136"
Nov 03 22:00:21.503875 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:21.503845    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p"
Nov 03 22:00:21.545670 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:21.545639    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f19698dc-e700-4e8e-a07c-7a5f9e74c70f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19698dc-e700-4e8e-a07c-7a5f9e74c70f-maas-api-service-ca\") pod \"f19698dc-e700-4e8e-a07c-7a5f9e74c70f\" (UID: \"f19698dc-e700-4e8e-a07c-7a5f9e74c70f\") "
Nov 03 22:00:21.545813 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:21.545738    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19698dc-e700-4e8e-a07c-7a5f9e74c70f-kube-api-access-7ddmc\" (UniqueName: \"kubernetes.io/projected/f19698dc-e700-4e8e-a07c-7a5f9e74c70f-kube-api-access-7ddmc\") pod \"f19698dc-e700-4e8e-a07c-7a5f9e74c70f\" (UID: \"f19698dc-e700-4e8e-a07c-7a5f9e74c70f\") "
Nov 03 22:00:21.546059 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:21.546028    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19698dc-e700-4e8e-a07c-7a5f9e74c70f-maas-api-service-ca" pod "f19698dc-e700-4e8e-a07c-7a5f9e74c70f" (UID: "f19698dc-e700-4e8e-a07c-7a5f9e74c70f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:21.548327 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:21.548307    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19698dc-e700-4e8e-a07c-7a5f9e74c70f-kube-api-access-7ddmc" pod "f19698dc-e700-4e8e-a07c-7a5f9e74c70f" (UID: "f19698dc-e700-4e8e-a07c-7a5f9e74c70f"). InnerVolumeSpecName "kube-api-access-7ddmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:21.647501 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:21.647466    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19698dc-e700-4e8e-a07c-7a5f9e74c70f-maas-api-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:21.647501 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:21.647501    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ddmc\" (UniqueName: \"kubernetes.io/projected/f19698dc-e700-4e8e-a07c-7a5f9e74c70f-kube-api-access-7ddmc\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:21.673032 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:21.672999    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1999e701-573c-454b-baca-bf466f9c2ced" path="/var/lib/kubelet/pods/1999e701-573c-454b-baca-bf466f9c2ced/volumes"
Nov 03 22:00:21.673487 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:21.673472    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbc6ade3-0fe6-45d5-a8a1-36118b848724" path="/var/lib/kubelet/pods/bbc6ade3-0fe6-45d5-a8a1-36118b848724/volumes"
Nov 03 22:00:21.673833 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:21.673821    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f07b51d2-c47b-400a-b786-4d500d736169" path="/var/lib/kubelet/pods/f07b51d2-c47b-400a-b786-4d500d736169/volumes"
Nov 03 22:00:21.686503 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:21.686482    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst"]
Nov 03 22:00:21.690717 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:21.690699    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895720-ghpst"]
Nov 03 22:00:22.360721 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:22.360690    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p"
Nov 03 22:00:22.361302 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:22.360726    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p" event={"ID":"f19698dc-e700-4e8e-a07c-7a5f9e74c70f","Type":"ContainerDied","Data":"c61c618567b569270c0ed178259fe25423dc774eb854685ef204ec2de0be787a"}
Nov 03 22:00:22.361302 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:22.360780    5053 scope.go:122] "RemoveContainer" containerID="cc12824bf999dee7be27bf28db21fc4ea8b73124f21b2aedf1fa5d30619accc0"
Nov 03 22:00:22.363642 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:22.363617    5053 generic.go:356] "Generic (PLEG): container finished" podID="a7e82a8c-058a-467f-870d-361ef4b86532" containerID="0d1f620d4e229a01fc761c02d028d1206ef841079ad91b74b652df8283e61bff" exitCode=0
Nov 03 22:00:22.363748 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:22.363640    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3112e9fa-68d69" event={"ID":"a7e82a8c-058a-467f-870d-361ef4b86532","Type":"ContainerDied","Data":"0d1f620d4e229a01fc761c02d028d1206ef841079ad91b74b652df8283e61bff"}
Nov 03 22:00:22.530144 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:22.530103    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p"]
Nov 03 22:00:22.577279 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:22.577189    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93c833-29895720-tt79p"]
Nov 03 22:00:23.545579 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:23.545552    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3112e9fa-68d69"
Nov 03 22:00:23.671299 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:23.671174    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53bee6ec-657e-49b0-9e45-2b1c4c1b275f" path="/var/lib/kubelet/pods/53bee6ec-657e-49b0-9e45-2b1c4c1b275f/volumes"
Nov 03 22:00:23.671807 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:23.671785    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f19698dc-e700-4e8e-a07c-7a5f9e74c70f" path="/var/lib/kubelet/pods/f19698dc-e700-4e8e-a07c-7a5f9e74c70f/volumes"
Nov 03 22:00:23.673795 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:23.673770    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7e82a8c-058a-467f-870d-361ef4b86532-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7e82a8c-058a-467f-870d-361ef4b86532-maas-api-xks-ca\") pod \"a7e82a8c-058a-467f-870d-361ef4b86532\" (UID: \"a7e82a8c-058a-467f-870d-361ef4b86532\") "
Nov 03 22:00:23.673904 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:23.673830    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7e82a8c-058a-467f-870d-361ef4b86532-kube-api-access-hg2tp\" (UniqueName: \"kubernetes.io/projected/a7e82a8c-058a-467f-870d-361ef4b86532-kube-api-access-hg2tp\") pod \"a7e82a8c-058a-467f-870d-361ef4b86532\" (UID: \"a7e82a8c-058a-467f-870d-361ef4b86532\") "
Nov 03 22:00:23.673904 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:23.673876    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7e82a8c-058a-467f-870d-361ef4b86532-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7e82a8c-058a-467f-870d-361ef4b86532-maas-api-openshift-ca\") pod \"a7e82a8c-058a-467f-870d-361ef4b86532\" (UID: \"a7e82a8c-058a-467f-870d-361ef4b86532\") "
Nov 03 22:00:23.674464 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:23.674356    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7e82a8c-058a-467f-870d-361ef4b86532-maas-api-openshift-ca" pod "a7e82a8c-058a-467f-870d-361ef4b86532" (UID: "a7e82a8c-058a-467f-870d-361ef4b86532"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:23.674594 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:23.674573    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7e82a8c-058a-467f-870d-361ef4b86532-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:23.676579 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:23.676561    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7e82a8c-058a-467f-870d-361ef4b86532-maas-api-xks-ca" pod "a7e82a8c-058a-467f-870d-361ef4b86532" (UID: "a7e82a8c-058a-467f-870d-361ef4b86532"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:00:23.677085 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:23.677061    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e82a8c-058a-467f-870d-361ef4b86532-kube-api-access-hg2tp" pod "a7e82a8c-058a-467f-870d-361ef4b86532" (UID: "a7e82a8c-058a-467f-870d-361ef4b86532"). InnerVolumeSpecName "kube-api-access-hg2tp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:23.775953 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:23.775922    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7e82a8c-058a-467f-870d-361ef4b86532-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:23.775953 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:23.775950    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg2tp\" (UniqueName: \"kubernetes.io/projected/a7e82a8c-058a-467f-870d-361ef4b86532-kube-api-access-hg2tp\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:24.380149 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:24.380106    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3112e9fa-68d69" event={"ID":"a7e82a8c-058a-467f-870d-361ef4b86532","Type":"ContainerDied","Data":"cc72af4093c35088d6e5bd6486efb0a4906cbae293f40957db228ba831a1a45b"}
Nov 03 22:00:24.380345 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:24.380156    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc72af4093c35088d6e5bd6486efb0a4906cbae293f40957db228ba831a1a45b"
Nov 03 22:00:24.380345 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:24.380155    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3112e9fa-68d69"
Nov 03 22:00:30.394645 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.394007    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"]
Nov 03 22:00:30.395072 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.394967    5053 cpu_manager.go:414] "containerMap: removing container" podUID="f19698dc-e700-4e8e-a07c-7a5f9e74c70f" containerName="cleanup"
Nov 03 22:00:30.395072 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.394990    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19698dc-e700-4e8e-a07c-7a5f9e74c70f" containerName="cleanup"
Nov 03 22:00:30.395072 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.395008    5053 cpu_manager.go:414] "containerMap: removing container" podUID="f19698dc-e700-4e8e-a07c-7a5f9e74c70f" containerName="cleanup"
Nov 03 22:00:30.395072 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.395016    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19698dc-e700-4e8e-a07c-7a5f9e74c70f" containerName="cleanup"
Nov 03 22:00:30.395072 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.395047    5053 cpu_manager.go:414] "containerMap: removing container" podUID="53bee6ec-657e-49b0-9e45-2b1c4c1b275f" containerName="cleanup"
Nov 03 22:00:30.395072 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.395055    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53bee6ec-657e-49b0-9e45-2b1c4c1b275f" containerName="cleanup"
Nov 03 22:00:30.395072 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.395068    5053 cpu_manager.go:414] "containerMap: removing container" podUID="f07b51d2-c47b-400a-b786-4d500d736169" containerName="cleanup"
Nov 03 22:00:30.395558 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.395077    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07b51d2-c47b-400a-b786-4d500d736169" containerName="cleanup"
Nov 03 22:00:30.395558 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.395101    5053 cpu_manager.go:414] "containerMap: removing container" podUID="bbc6ade3-0fe6-45d5-a8a1-36118b848724" containerName="cleanup"
Nov 03 22:00:30.395558 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.395108    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc6ade3-0fe6-45d5-a8a1-36118b848724" containerName="cleanup"
Nov 03 22:00:30.395558 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.395121    5053 cpu_manager.go:414] "containerMap: removing container" podUID="1999e701-573c-454b-baca-bf466f9c2ced" containerName="cleanup"
Nov 03 22:00:30.395558 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.395128    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1999e701-573c-454b-baca-bf466f9c2ced" containerName="cleanup"
Nov 03 22:00:30.395558 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.395142    5053 cpu_manager.go:414] "containerMap: removing container" podUID="53bee6ec-657e-49b0-9e45-2b1c4c1b275f" containerName="cleanup"
Nov 03 22:00:30.395558 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.395148    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53bee6ec-657e-49b0-9e45-2b1c4c1b275f" containerName="cleanup"
Nov 03 22:00:30.395558 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.395167    5053 cpu_manager.go:414] "containerMap: removing container" podUID="53bee6ec-657e-49b0-9e45-2b1c4c1b275f" containerName="cleanup"
Nov 03 22:00:30.395558 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.395174    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53bee6ec-657e-49b0-9e45-2b1c4c1b275f" containerName="cleanup"
Nov 03 22:00:30.395558 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.395188    5053 cpu_manager.go:414] "containerMap: removing container" podUID="a7e82a8c-058a-467f-870d-361ef4b86532" containerName="revoke-keys"
Nov 03 22:00:30.395558 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.395195    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e82a8c-058a-467f-870d-361ef4b86532" containerName="revoke-keys"
Nov 03 22:00:30.395558 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.395367    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="1999e701-573c-454b-baca-bf466f9c2ced" containerName="cleanup"
Nov 03 22:00:30.395558 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.395383    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="53bee6ec-657e-49b0-9e45-2b1c4c1b275f" containerName="cleanup"
Nov 03 22:00:30.395558 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.395391    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="f07b51d2-c47b-400a-b786-4d500d736169" containerName="cleanup"
Nov 03 22:00:30.395558 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.395401    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc6ade3-0fe6-45d5-a8a1-36118b848724" containerName="cleanup"
Nov 03 22:00:30.395558 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.395414    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="53bee6ec-657e-49b0-9e45-2b1c4c1b275f" containerName="cleanup"
Nov 03 22:00:30.395558 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.395423    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e82a8c-058a-467f-870d-361ef4b86532" containerName="revoke-keys"
Nov 03 22:00:30.395558 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.395436    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19698dc-e700-4e8e-a07c-7a5f9e74c70f" containerName="cleanup"
Nov 03 22:00:30.395558 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.395445    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19698dc-e700-4e8e-a07c-7a5f9e74c70f" containerName="cleanup"
Nov 03 22:00:30.400972 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.400952    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.469260 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.469198    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.469478 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.469333    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.469478 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.469396    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.469478 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.469434    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.470429 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.470395    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.470707 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.470442    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6t8m\" (UniqueName: \"kubernetes.io/projected/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-kube-api-access-t6t8m\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.470707 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.470585    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.470861 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.470728    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.470861 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.470747    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.492000 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.491945    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"]
Nov 03 22:00:30.572046 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.572008    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.572046 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.572049    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.572375 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.572069    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.572375 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.572090    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.572375 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.572128    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.572375 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.572153    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.572375 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.572184    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.572375 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.572215    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6t8m\" (UniqueName: \"kubernetes.io/projected/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-kube-api-access-t6t8m\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.572375 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.572277    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.572760 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.572552    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.572893 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.572856    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.573027 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.572923    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.573027 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.572978    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.573344 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.573043    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.574661 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.574641    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.574863 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.574846    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.628800 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.628767    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.628953 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.628830    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6t8m\" (UniqueName: \"kubernetes.io/projected/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-kube-api-access-t6t8m\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:30.716686 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:30.716655    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:31.183006 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:31.182970    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"]
Nov 03 22:00:31.183288 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:00:31.183259    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee6091e_7d0a_43b8_836e_9565f25ecc5b.slice/crio-06ef4261f9ab3e82117582744e14bf288683e6b4eaf4e12b12c5d5d46ef43b2a WatchSource:0}: Error finding container 06ef4261f9ab3e82117582744e14bf288683e6b4eaf4e12b12c5d5d46ef43b2a: Status 404 returned error can't find the container with id 06ef4261f9ab3e82117582744e14bf288683e6b4eaf4e12b12c5d5d46ef43b2a
Nov 03 22:00:31.185728 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:31.185700    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:00:31.185774 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:31.185760    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:00:31.185811 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:31.185790    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:00:31.431768 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:31.431674    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b" event={"ID":"7ee6091e-7d0a-43b8-836e-9565f25ecc5b","Type":"ContainerStarted","Data":"defba414ce9da88bb2955f57004a36450d31176b874736a7dac967e61c069867"}
Nov 03 22:00:31.431768 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:31.431759    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b" event={"ID":"7ee6091e-7d0a-43b8-836e-9565f25ecc5b","Type":"ContainerStarted","Data":"06ef4261f9ab3e82117582744e14bf288683e6b4eaf4e12b12c5d5d46ef43b2a"}
Nov 03 22:00:31.551054 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:31.550991    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b" podStartSLOduration=1.5509727660000001 podStartE2EDuration="1.550972766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:31.549365912 +0000 UTC m=+2746.043287779" watchObservedRunningTime="2026-11-03 22:00:31.550972766 +0000 UTC m=+2746.044894643"
Nov 03 22:00:31.717312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:31.717269    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:32.725263 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:32.724371    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 03 22:00:32.725263 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:32.724438    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b" podUID="7ee6091e-7d0a-43b8-836e-9565f25ecc5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 03 22:00:33.717816 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:33.717780    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 03 22:00:33.718114 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:33.718084    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b" podUID="7ee6091e-7d0a-43b8-836e-9565f25ecc5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 03 22:00:34.717707 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:34.717665    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 03 22:00:34.718358 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:34.718325    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b" podUID="7ee6091e-7d0a-43b8-836e-9565f25ecc5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 03 22:00:35.717626 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:35.717589    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 03 22:00:35.717940 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:35.717909    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b" podUID="7ee6091e-7d0a-43b8-836e-9565f25ecc5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 03 22:00:36.718674 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:36.718632    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 03 22:00:36.719282 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:36.719226    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b" podUID="7ee6091e-7d0a-43b8-836e-9565f25ecc5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 03 22:00:37.731629 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:37.731591    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 03 22:00:37.732189 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:37.732157    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b" podUID="7ee6091e-7d0a-43b8-836e-9565f25ecc5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 03 22:00:38.772329 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:38.772296    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:38.772821 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:38.772781    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:38.774095 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:38.774062    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:00:39.016409 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.016362    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"]
Nov 03 22:00:39.017305 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.017278    5053 cpu_manager.go:414] "containerMap: removing container" podUID="f19698dc-e700-4e8e-a07c-7a5f9e74c70f" containerName="cleanup"
Nov 03 22:00:39.017367 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.017309    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19698dc-e700-4e8e-a07c-7a5f9e74c70f" containerName="cleanup"
Nov 03 22:00:39.017514 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.017503    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19698dc-e700-4e8e-a07c-7a5f9e74c70f" containerName="cleanup"
Nov 03 22:00:39.017546 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.017529    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="53bee6ec-657e-49b0-9e45-2b1c4c1b275f" containerName="cleanup"
Nov 03 22:00:39.023021 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.022959    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.075922 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.075884    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7ee809-openshift-default-dockercfg-qsllw\""
Nov 03 22:00:39.077228 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.077201    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"]
Nov 03 22:00:39.117627 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.116907    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.117627 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.116965    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.117627 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.117015    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a891ded-a561-4750-8f74-e85eed3f825f-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.117627 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.117065    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.117627 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.117094    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a891ded-a561-4750-8f74-e85eed3f825f-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.117627 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.117158    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.117627 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.117230    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v47t\" (UniqueName: \"kubernetes.io/projected/9a891ded-a561-4750-8f74-e85eed3f825f-kube-api-access-7v47t\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.117627 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.117338    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.117627 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.117384    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a891ded-a561-4750-8f74-e85eed3f825f-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.218528 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.218491    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v47t\" (UniqueName: \"kubernetes.io/projected/9a891ded-a561-4750-8f74-e85eed3f825f-kube-api-access-7v47t\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.218733 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.218564    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.218733 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.218593    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a891ded-a561-4750-8f74-e85eed3f825f-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.218733 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.218664    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.218733 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.218683    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.218733 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.218723    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a891ded-a561-4750-8f74-e85eed3f825f-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.219028 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.218758    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.219028 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.218783    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a891ded-a561-4750-8f74-e85eed3f825f-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.219028 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.218846    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.219193 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.219131    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.219193 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.219178    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.219442 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.219387    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.219747 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.219722    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a891ded-a561-4750-8f74-e85eed3f825f-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.219837 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.219788    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.221434 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.221405    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a891ded-a561-4750-8f74-e85eed3f825f-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.221531 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.221447    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.286833 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.286757    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v47t\" (UniqueName: \"kubernetes.io/projected/9a891ded-a561-4750-8f74-e85eed3f825f-kube-api-access-7v47t\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.290916 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.290883    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a891ded-a561-4750-8f74-e85eed3f825f-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.338218 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.338184    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:39.763903 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.763873    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"]
Nov 03 22:00:39.765013 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:00:39.764989    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a891ded_a561_4750_8f74_e85eed3f825f.slice/crio-66e19a45ff58f9d7e7777001da745834e67197e2eb87505d14d0f47487f6f995 WatchSource:0}: Error finding container 66e19a45ff58f9d7e7777001da745834e67197e2eb87505d14d0f47487f6f995: Status 404 returned error can't find the container with id 66e19a45ff58f9d7e7777001da745834e67197e2eb87505d14d0f47487f6f995
Nov 03 22:00:39.767538 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.767507    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:00:39.767677 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.767587    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:00:39.767677 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:39.767619    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:00:40.523719 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:40.523680    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv" event={"ID":"9a891ded-a561-4750-8f74-e85eed3f825f","Type":"ContainerStarted","Data":"c39e7735547ea83b6b69b945c601fb97812241b24fd78769e93a1f998a0608a9"}
Nov 03 22:00:40.524319 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:40.524294    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv" event={"ID":"9a891ded-a561-4750-8f74-e85eed3f825f","Type":"ContainerStarted","Data":"66e19a45ff58f9d7e7777001da745834e67197e2eb87505d14d0f47487f6f995"}
Nov 03 22:00:40.640423 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:40.640370    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv" podStartSLOduration=2.640346138 podStartE2EDuration="2.640346138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:40.639446149 +0000 UTC m=+2755.133368025" watchObservedRunningTime="2026-11-03 22:00:40.640346138 +0000 UTC m=+2755.134268084"
Nov 03 22:00:40.953483 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:00:40.953440    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:00:41.339374 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:41.339294    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:41.340975 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:41.340945    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 03 22:00:41.341134 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:41.341036    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv" podUID="9a891ded-a561-4750-8f74-e85eed3f825f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 03 22:00:42.339204 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:42.339169    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 03 22:00:42.339670 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:42.339229    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv" podUID="9a891ded-a561-4750-8f74-e85eed3f825f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 03 22:00:43.339008 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:43.338974    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 03 22:00:43.339299 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:43.339262    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv" podUID="9a891ded-a561-4750-8f74-e85eed3f825f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 03 22:00:44.338947 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:44.338909    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 03 22:00:44.339148 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:44.338982    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv" podUID="9a891ded-a561-4750-8f74-e85eed3f825f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 03 22:00:45.338878 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:45.338844    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 03 22:00:45.339338 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:45.338898    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv" podUID="9a891ded-a561-4750-8f74-e85eed3f825f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 03 22:00:46.339174 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:46.339138    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 03 22:00:46.339686 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:46.339210    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv" podUID="9a891ded-a561-4750-8f74-e85eed3f825f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 03 22:00:47.343485 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:47.343454    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:47.343937 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:47.343924    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:00:47.344674 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:00:47.344656    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:01:09.786113 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:09.786080    5053 generic.go:356] "Generic (PLEG): container finished" podID="b8574f16-35c1-4717-a83a-396210ca7a1f" containerID="78cddba9c25d4cf6bb570f4c9e894c3c520919fe645ef8a1eca28f9e45fe8ed1" exitCode=0
Nov 03 22:01:09.786607 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:09.786159    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g" event={"ID":"b8574f16-35c1-4717-a83a-396210ca7a1f","Type":"ContainerDied","Data":"78cddba9c25d4cf6bb570f4c9e894c3c520919fe645ef8a1eca28f9e45fe8ed1"}
Nov 03 22:01:09.786607 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:09.786585    5053 scope.go:122] "RemoveContainer" containerID="78cddba9c25d4cf6bb570f4c9e894c3c520919fe645ef8a1eca28f9e45fe8ed1"
Nov 03 22:01:09.788150 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:09.788124    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:01:09.788226 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:09.788190    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:01:09.788226 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:09.788223    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:01:10.794539 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:10.794507    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g" event={"ID":"b8574f16-35c1-4717-a83a-396210ca7a1f","Type":"ContainerStarted","Data":"66a8879dd6a4aa2f725f314ebc129836d6ebd079e40ea6319901894390e4ff69"}
Nov 03 22:01:10.953193 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:01:10.953158    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:01:11.027340 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:11.027300    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 22:01:11.028919 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:11.028891    5053 patch_prober.go:29] interesting pod/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 03 22:01:11.029055 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:11.028936    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g" podUID="b8574f16-35c1-4717-a83a-396210ca7a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 03 22:01:12.027747 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:12.027708    5053 patch_prober.go:29] interesting pod/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 03 22:01:12.028407 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:12.028368    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g" podUID="b8574f16-35c1-4717-a83a-396210ca7a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 03 22:01:13.027703 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:13.027665    5053 patch_prober.go:29] interesting pod/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 03 22:01:13.027883 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:13.027734    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g" podUID="b8574f16-35c1-4717-a83a-396210ca7a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 03 22:01:14.027796 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:14.027756    5053 patch_prober.go:29] interesting pod/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 03 22:01:14.028002 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:14.027829    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g" podUID="b8574f16-35c1-4717-a83a-396210ca7a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 03 22:01:14.714262 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:14.711340    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 22:01:15.027703 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:15.027608    5053 patch_prober.go:29] interesting pod/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 03 22:01:15.027974 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:15.027947    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g" podUID="b8574f16-35c1-4717-a83a-396210ca7a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 03 22:01:16.032405 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:16.032370    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 22:01:16.033724 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:16.033694    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 22:01:35.048230 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:35.048190    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"]
Nov 03 22:01:35.052842 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:35.052823    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"
Nov 03 22:01:35.108137 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:35.108103    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"]
Nov 03 22:01:35.152603 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:35.152570    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-5bd35c-serving-cert\""
Nov 03 22:01:35.161199 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:35.161169    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed429be0-1f3f-4461-94d3-c4f18c389676-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22\" (UID: \"ed429be0-1f3f-4461-94d3-c4f18c389676\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"
Nov 03 22:01:35.161353 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:35.161340    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed429be0-1f3f-4461-94d3-c4f18c389676-metrics-certs\") pod \"maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22\" (UID: \"ed429be0-1f3f-4461-94d3-c4f18c389676\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"
Nov 03 22:01:35.161427 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:35.161409    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96hxs\" (UniqueName: \"kubernetes.io/projected/ed429be0-1f3f-4461-94d3-c4f18c389676-kube-api-access-96hxs\") pod \"maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22\" (UID: \"ed429be0-1f3f-4461-94d3-c4f18c389676\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"
Nov 03 22:01:35.161565 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:35.161552    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed429be0-1f3f-4461-94d3-c4f18c389676-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22\" (UID: \"ed429be0-1f3f-4461-94d3-c4f18c389676\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"
Nov 03 22:01:35.262444 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:35.262404    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed429be0-1f3f-4461-94d3-c4f18c389676-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22\" (UID: \"ed429be0-1f3f-4461-94d3-c4f18c389676\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"
Nov 03 22:01:35.262651 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:35.262484    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed429be0-1f3f-4461-94d3-c4f18c389676-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22\" (UID: \"ed429be0-1f3f-4461-94d3-c4f18c389676\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"
Nov 03 22:01:35.262651 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:35.262565    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed429be0-1f3f-4461-94d3-c4f18c389676-metrics-certs\") pod \"maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22\" (UID: \"ed429be0-1f3f-4461-94d3-c4f18c389676\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"
Nov 03 22:01:35.262651 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:35.262607    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96hxs\" (UniqueName: \"kubernetes.io/projected/ed429be0-1f3f-4461-94d3-c4f18c389676-kube-api-access-96hxs\") pod \"maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22\" (UID: \"ed429be0-1f3f-4461-94d3-c4f18c389676\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"
Nov 03 22:01:35.262651 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:01:35.262641    5053 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-serving-cert: secret "maas-api-e2e-praxis-b-5bd35c-serving-cert" not found
Nov 03 22:01:35.262899 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:01:35.262731    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed429be0-1f3f-4461-94d3-c4f18c389676-maas-api-tls podName:ed429be0-1f3f-4461-94d3-c4f18c389676 nodeName:}" failed. No retries permitted until 2026-11-03 22:01:35.762711364 +0000 UTC m=+2810.256633220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ed429be0-1f3f-4461-94d3-c4f18c389676-maas-api-tls") pod "maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22" (UID: "ed429be0-1f3f-4461-94d3-c4f18c389676") : secret "maas-api-e2e-praxis-b-5bd35c-serving-cert" not found
Nov 03 22:01:35.263154 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:35.263127    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed429be0-1f3f-4461-94d3-c4f18c389676-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22\" (UID: \"ed429be0-1f3f-4461-94d3-c4f18c389676\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"
Nov 03 22:01:35.265161 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:35.265142    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed429be0-1f3f-4461-94d3-c4f18c389676-metrics-certs\") pod \"maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22\" (UID: \"ed429be0-1f3f-4461-94d3-c4f18c389676\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"
Nov 03 22:01:35.323153 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:35.323074    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96hxs\" (UniqueName: \"kubernetes.io/projected/ed429be0-1f3f-4461-94d3-c4f18c389676-kube-api-access-96hxs\") pod \"maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22\" (UID: \"ed429be0-1f3f-4461-94d3-c4f18c389676\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"
Nov 03 22:01:35.771872 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:35.771839    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed429be0-1f3f-4461-94d3-c4f18c389676-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22\" (UID: \"ed429be0-1f3f-4461-94d3-c4f18c389676\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"
Nov 03 22:01:35.774390 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:35.774370    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed429be0-1f3f-4461-94d3-c4f18c389676-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22\" (UID: \"ed429be0-1f3f-4461-94d3-c4f18c389676\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"
Nov 03 22:01:35.966204 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:35.966169    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"
Nov 03 22:01:36.346251 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:36.346206    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"]
Nov 03 22:01:36.347135 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:01:36.347103    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded429be0_1f3f_4461_94d3_c4f18c389676.slice/crio-84a87641be8a4effab771508f5f617e4e6c2a9f1cec4cf5c587181f62af94da1 WatchSource:0}: Error finding container 84a87641be8a4effab771508f5f617e4e6c2a9f1cec4cf5c587181f62af94da1: Status 404 returned error can't find the container with id 84a87641be8a4effab771508f5f617e4e6c2a9f1cec4cf5c587181f62af94da1
Nov 03 22:01:37.017082 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:37.017040    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22" event={"ID":"ed429be0-1f3f-4461-94d3-c4f18c389676","Type":"ContainerStarted","Data":"84a87641be8a4effab771508f5f617e4e6c2a9f1cec4cf5c587181f62af94da1"}
Nov 03 22:01:38.026042 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:38.026005    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22" event={"ID":"ed429be0-1f3f-4461-94d3-c4f18c389676","Type":"ContainerStarted","Data":"c6660f18f8b895525a0de0581418df4f731b3fd157f92848b00c66494ff06776"}
Nov 03 22:01:38.026574 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:38.026213    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"
Nov 03 22:01:38.141044 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:38.140984    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22" podStartSLOduration=2.357125499 podStartE2EDuration="3.140968184s" totalImagesPullingTime="783.842685ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:01:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:01:38.139974315 +0000 UTC m=+2812.633896172" watchObservedRunningTime="2026-11-03 22:01:38.140968184 +0000 UTC m=+2812.634890057"
Nov 03 22:01:40.932679 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:01:40.932636    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:01:44.041837 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:44.041804    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"
Nov 03 22:01:45.972371 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:45.972337    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"]
Nov 03 22:01:45.976569 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:45.976547    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.068385 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.068347    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"]
Nov 03 22:01:46.120738 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.120701    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0626b947-6a73-4669-b426-b5b9ce1c30e3-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.120971 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.120951    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.121154 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.121133    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.121358 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.121169    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sk44\" (UniqueName: \"kubernetes.io/projected/0626b947-6a73-4669-b426-b5b9ce1c30e3-kube-api-access-9sk44\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.121358 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.121213    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.121358 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.121313    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.121358 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.121350    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.121527 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.121379    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.121527 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.121408    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.223393 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.223294    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.223393 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.223379    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.223641 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.223423    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.223641 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.223467    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.223641 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.223526    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0626b947-6a73-4669-b426-b5b9ce1c30e3-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.223641 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.223565    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.223641 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.223633    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.223912 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.223681    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sk44\" (UniqueName: \"kubernetes.io/projected/0626b947-6a73-4669-b426-b5b9ce1c30e3-kube-api-access-9sk44\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.223912 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.223747    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.223912 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.223776    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.224064 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.223974    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.224117 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.224102    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.224299 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.224272    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.224661 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.224642    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0626b947-6a73-4669-b426-b5b9ce1c30e3-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.226487 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.226463    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.226689 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.226666    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.288178 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.288147    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.288375 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.288206    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sk44\" (UniqueName: \"kubernetes.io/projected/0626b947-6a73-4669-b426-b5b9ce1c30e3-kube-api-access-9sk44\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.290078 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.290060    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:46.729982 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.729948    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"]
Nov 03 22:01:46.769067 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:01:46.769036    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0626b947_6a73_4669_b426_b5b9ce1c30e3.slice/crio-cb491dfea912d24796751ef8759652ec8f92a2b2078604ab8f87a421585a61ea WatchSource:0}: Error finding container cb491dfea912d24796751ef8759652ec8f92a2b2078604ab8f87a421585a61ea: Status 404 returned error can't find the container with id cb491dfea912d24796751ef8759652ec8f92a2b2078604ab8f87a421585a61ea
Nov 03 22:01:46.771801 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.771766    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:01:46.771904 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.771837    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:01:46.771904 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:46.771866    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:01:47.109983 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:47.109894    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh" event={"ID":"0626b947-6a73-4669-b426-b5b9ce1c30e3","Type":"ContainerStarted","Data":"3e1f6328dcc2a3a03c4cdbae3a79687526a8864535f3ca532cc932acae33829c"}
Nov 03 22:01:47.109983 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:47.109959    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh" event={"ID":"0626b947-6a73-4669-b426-b5b9ce1c30e3","Type":"ContainerStarted","Data":"cb491dfea912d24796751ef8759652ec8f92a2b2078604ab8f87a421585a61ea"}
Nov 03 22:01:47.294000 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:47.293615    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:47.299650 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:47.299607    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 03 22:01:47.299867 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:47.299683    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh" podUID="0626b947-6a73-4669-b426-b5b9ce1c30e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 03 22:01:48.290872 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:48.290837    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 03 22:01:48.291333 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:48.290899    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh" podUID="0626b947-6a73-4669-b426-b5b9ce1c30e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 03 22:01:49.291262 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:49.291211    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 03 22:01:49.291672 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:49.291278    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh" podUID="0626b947-6a73-4669-b426-b5b9ce1c30e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 03 22:01:50.291038 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:50.290999    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 03 22:01:50.291229 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:50.291059    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh" podUID="0626b947-6a73-4669-b426-b5b9ce1c30e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 03 22:01:51.291358 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:51.291313    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 03 22:01:51.291851 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:51.291374    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh" podUID="0626b947-6a73-4669-b426-b5b9ce1c30e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 03 22:01:52.290591 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:52.290551    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 03 22:01:52.290778 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:52.290617    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh" podUID="0626b947-6a73-4669-b426-b5b9ce1c30e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 03 22:01:53.291329 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:53.291289    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 03 22:01:53.291792 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:53.291358    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh" podUID="0626b947-6a73-4669-b426-b5b9ce1c30e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 03 22:01:55.154880 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:55.154832    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:55.155422 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:55.155203    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:55.156307 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:55.156283    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:01:55.338149 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:01:55.338093    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh" podStartSLOduration=10.33807584 podStartE2EDuration="10.33807584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:01:47.227105996 +0000 UTC m=+2821.721027872" watchObservedRunningTime="2026-11-03 22:01:55.33807584 +0000 UTC m=+2829.831997714"
Nov 03 22:02:10.930343 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:02:10.930303    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:02:27.047259 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:27.047210    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b2a8f46-bfh7k"]
Nov 03 22:02:27.095782 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:27.095749    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b2a8f46-bfh7k"]
Nov 03 22:02:27.668653 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:27.668615    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="018a254b-8a68-402c-a0d2-ce1fbf23796f" path="/var/lib/kubelet/pods/018a254b-8a68-402c-a0d2-ce1fbf23796f/volumes"
Nov 03 22:02:34.457527 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:34.457492    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"]
Nov 03 22:02:34.457962 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:34.457713    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l" podUID="34ee5ddf-3c64-4a43-962f-834963d7452a" containerName="maas-api" containerID="cri-o://9ecc5675229be308c9993b599d9487d606136ebfb5036961609b1156bcd1c7f5" gracePeriod=30
Nov 03 22:02:34.764105 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:34.764076    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"
Nov 03 22:02:34.859065 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:34.859030    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34ee5ddf-3c64-4a43-962f-834963d7452a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/34ee5ddf-3c64-4a43-962f-834963d7452a-metrics-certs\") pod \"34ee5ddf-3c64-4a43-962f-834963d7452a\" (UID: \"34ee5ddf-3c64-4a43-962f-834963d7452a\") "
Nov 03 22:02:34.859280 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:34.859139    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34ee5ddf-3c64-4a43-962f-834963d7452a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34ee5ddf-3c64-4a43-962f-834963d7452a-maas-api-tls\") pod \"34ee5ddf-3c64-4a43-962f-834963d7452a\" (UID: \"34ee5ddf-3c64-4a43-962f-834963d7452a\") "
Nov 03 22:02:34.859280 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:34.859176    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34ee5ddf-3c64-4a43-962f-834963d7452a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ee5ddf-3c64-4a43-962f-834963d7452a-openshift-service-ca\") pod \"34ee5ddf-3c64-4a43-962f-834963d7452a\" (UID: \"34ee5ddf-3c64-4a43-962f-834963d7452a\") "
Nov 03 22:02:34.859280 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:34.859254    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34ee5ddf-3c64-4a43-962f-834963d7452a-kube-api-access-bx82l\" (UniqueName: \"kubernetes.io/projected/34ee5ddf-3c64-4a43-962f-834963d7452a-kube-api-access-bx82l\") pod \"34ee5ddf-3c64-4a43-962f-834963d7452a\" (UID: \"34ee5ddf-3c64-4a43-962f-834963d7452a\") "
Nov 03 22:02:34.859577 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:34.859552    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ee5ddf-3c64-4a43-962f-834963d7452a-openshift-service-ca" pod "34ee5ddf-3c64-4a43-962f-834963d7452a" (UID: "34ee5ddf-3c64-4a43-962f-834963d7452a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:02:34.859982 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:34.859888    5053 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ee5ddf-3c64-4a43-962f-834963d7452a-openshift-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:34.861705 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:34.861683    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34ee5ddf-3c64-4a43-962f-834963d7452a-maas-api-tls" pod "34ee5ddf-3c64-4a43-962f-834963d7452a" (UID: "34ee5ddf-3c64-4a43-962f-834963d7452a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:02:34.862016 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:34.861999    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ee5ddf-3c64-4a43-962f-834963d7452a-kube-api-access-bx82l" pod "34ee5ddf-3c64-4a43-962f-834963d7452a" (UID: "34ee5ddf-3c64-4a43-962f-834963d7452a"). InnerVolumeSpecName "kube-api-access-bx82l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:02:34.862016 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:34.862000    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34ee5ddf-3c64-4a43-962f-834963d7452a-metrics-certs" pod "34ee5ddf-3c64-4a43-962f-834963d7452a" (UID: "34ee5ddf-3c64-4a43-962f-834963d7452a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:02:34.961220 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:34.961169    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx82l\" (UniqueName: \"kubernetes.io/projected/34ee5ddf-3c64-4a43-962f-834963d7452a-kube-api-access-bx82l\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:34.961220 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:34.961206    5053 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34ee5ddf-3c64-4a43-962f-834963d7452a-metrics-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:34.961220 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:34.961216    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34ee5ddf-3c64-4a43-962f-834963d7452a-maas-api-tls\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:35.574151 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:35.574114    5053 generic.go:356] "Generic (PLEG): container finished" podID="34ee5ddf-3c64-4a43-962f-834963d7452a" containerID="9ecc5675229be308c9993b599d9487d606136ebfb5036961609b1156bcd1c7f5" exitCode=0
Nov 03 22:02:35.574662 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:35.574178    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"
Nov 03 22:02:35.574662 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:35.574178    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l" event={"ID":"34ee5ddf-3c64-4a43-962f-834963d7452a","Type":"ContainerDied","Data":"9ecc5675229be308c9993b599d9487d606136ebfb5036961609b1156bcd1c7f5"}
Nov 03 22:02:35.574662 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:35.574301    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l" event={"ID":"34ee5ddf-3c64-4a43-962f-834963d7452a","Type":"ContainerDied","Data":"2b8f8021527a92f3ddbfe68a61f592ebb547bac741867465be9797184d1fc23e"}
Nov 03 22:02:35.574662 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:35.574333    5053 scope.go:122] "RemoveContainer" containerID="9ecc5675229be308c9993b599d9487d606136ebfb5036961609b1156bcd1c7f5"
Nov 03 22:02:35.588894 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:35.588854    5053 scope.go:122] "RemoveContainer" containerID="9ecc5675229be308c9993b599d9487d606136ebfb5036961609b1156bcd1c7f5"
Nov 03 22:02:35.589223 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:02:35.589200    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ecc5675229be308c9993b599d9487d606136ebfb5036961609b1156bcd1c7f5\": container with ID starting with 9ecc5675229be308c9993b599d9487d606136ebfb5036961609b1156bcd1c7f5 not found: ID does not exist" containerID="9ecc5675229be308c9993b599d9487d606136ebfb5036961609b1156bcd1c7f5"
Nov 03 22:02:35.589352 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:35.589253    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ecc5675229be308c9993b599d9487d606136ebfb5036961609b1156bcd1c7f5"} err="failed to get container status \"9ecc5675229be308c9993b599d9487d606136ebfb5036961609b1156bcd1c7f5\": rpc error: code = NotFound desc = could not find container \"9ecc5675229be308c9993b599d9487d606136ebfb5036961609b1156bcd1c7f5\": container with ID starting with 9ecc5675229be308c9993b599d9487d606136ebfb5036961609b1156bcd1c7f5 not found: ID does not exist"
Nov 03 22:02:35.738639 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:35.738582    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"]
Nov 03 22:02:35.788575 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:35.788543    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c833-5f5d8c6bb4-vzg7l"]
Nov 03 22:02:37.669061 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:37.669020    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34ee5ddf-3c64-4a43-962f-834963d7452a" path="/var/lib/kubelet/pods/34ee5ddf-3c64-4a43-962f-834963d7452a/volumes"
Nov 03 22:02:39.956199 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:39.956161    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"]
Nov 03 22:02:39.957048 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:39.957029    5053 cpu_manager.go:414] "containerMap: removing container" podUID="34ee5ddf-3c64-4a43-962f-834963d7452a" containerName="maas-api"
Nov 03 22:02:39.957107 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:39.957052    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34ee5ddf-3c64-4a43-962f-834963d7452a" containerName="maas-api"
Nov 03 22:02:39.957156 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:39.957147    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="34ee5ddf-3c64-4a43-962f-834963d7452a" containerName="maas-api"
Nov 03 22:02:39.962293 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:39.962270    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.006129 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.006089    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"]
Nov 03 22:02:40.010408 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.010386    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.061006 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.060973    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"]
Nov 03 22:02:40.120641 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.112879    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"]
Nov 03 22:02:40.158780 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.158743    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.159003 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.158879    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt6jf\" (UniqueName: \"kubernetes.io/projected/978944c9-d91e-493f-8769-d6dacb9c679d-kube-api-access-gt6jf\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.159003 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.158916    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.159140 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.159012    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/978944c9-d91e-493f-8769-d6dacb9c679d-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.159140 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.159043    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.159282 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.159152    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.159282 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.159175    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/978944c9-d91e-493f-8769-d6dacb9c679d-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.159282 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.159199    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/978944c9-d91e-493f-8769-d6dacb9c679d-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.159456 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.159320    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.159456 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.159358    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.159456 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.159385    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.159680 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.159462    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.159680 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.159480    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.159680 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.159505    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb5cb186-a9c9-491b-8798-77e9cb223d27-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.159680 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.159556    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.159680 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.159572    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.159680 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.159601    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.159680 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.159615    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds8ph\" (UniqueName: \"kubernetes.io/projected/cb5cb186-a9c9-491b-8798-77e9cb223d27-kube-api-access-ds8ph\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.260523 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.260440    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.260523 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.260473    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.260523 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.260494    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.260523 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.260510    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds8ph\" (UniqueName: \"kubernetes.io/projected/cb5cb186-a9c9-491b-8798-77e9cb223d27-kube-api-access-ds8ph\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.260921 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.260546    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.260921 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.260640    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt6jf\" (UniqueName: \"kubernetes.io/projected/978944c9-d91e-493f-8769-d6dacb9c679d-kube-api-access-gt6jf\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.260921 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.260674    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.260921 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.260749    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/978944c9-d91e-493f-8769-d6dacb9c679d-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.260921 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.260788    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.260921 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.260860    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.260921 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.260891    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/978944c9-d91e-493f-8769-d6dacb9c679d-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.260921 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.260916    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/978944c9-d91e-493f-8769-d6dacb9c679d-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.261451 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.260953    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.261451 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.260976    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.261451 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.261029    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.261451 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.261082    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.261451 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.261149    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.261451 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.261178    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.261451 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.261219    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb5cb186-a9c9-491b-8798-77e9cb223d27-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.261451 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.261259    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.261451 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.260958    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.261451 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.261335    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.261955 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.261495    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.261955 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.261865    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.264371 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.262100    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.264371 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.262554    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb5cb186-a9c9-491b-8798-77e9cb223d27-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.264371 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.262556    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.264371 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.262765    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/978944c9-d91e-493f-8769-d6dacb9c679d-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.264688 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.264413    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.265751 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.265716    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.266034 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.266012    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/978944c9-d91e-493f-8769-d6dacb9c679d-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.274551 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.274525    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.317354 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.317316    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.317354 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.317339    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds8ph\" (UniqueName: \"kubernetes.io/projected/cb5cb186-a9c9-491b-8798-77e9cb223d27-kube-api-access-ds8ph\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.317624 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.317569    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/978944c9-d91e-493f-8769-d6dacb9c679d-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.317871 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.317850    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt6jf\" (UniqueName: \"kubernetes.io/projected/978944c9-d91e-493f-8769-d6dacb9c679d-kube-api-access-gt6jf\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.324842 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.324818    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:40.577273 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.577168    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:40.755544 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.755519    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"]
Nov 03 22:02:40.755740 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:02:40.755717    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5cb186_a9c9_491b_8798_77e9cb223d27.slice/crio-f86ba2f49835eee7ec41330a2d4542c95029c5240f713603d15aafe02a57755c WatchSource:0}: Error finding container f86ba2f49835eee7ec41330a2d4542c95029c5240f713603d15aafe02a57755c: Status 404 returned error can't find the container with id f86ba2f49835eee7ec41330a2d4542c95029c5240f713603d15aafe02a57755c
Nov 03 22:02:40.758410 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.758377    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:02:40.758501 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.758440    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:02:40.758501 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.758467    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:02:40.931150 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:02:40.931111    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:02:40.960003 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.959975    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"]
Nov 03 22:02:40.961534 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:02:40.961506    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978944c9_d91e_493f_8769_d6dacb9c679d.slice/crio-0ec2cdc63d0a517ab2077b6be9e6f3a01352d3246dd2c4546d3ea9ce6d6b26c1 WatchSource:0}: Error finding container 0ec2cdc63d0a517ab2077b6be9e6f3a01352d3246dd2c4546d3ea9ce6d6b26c1: Status 404 returned error can't find the container with id 0ec2cdc63d0a517ab2077b6be9e6f3a01352d3246dd2c4546d3ea9ce6d6b26c1
Nov 03 22:02:40.964476 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.964436    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:02:40.964568 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.964526    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:02:40.964568 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:40.964561    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:02:41.641102 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:41.641061    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2" event={"ID":"978944c9-d91e-493f-8769-d6dacb9c679d","Type":"ContainerStarted","Data":"e6a38db1891311fd8358762e45bcd78549d4ccc5ef2ab9800ae8551e8433fc85"}
Nov 03 22:02:41.641334 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:41.641134    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2" event={"ID":"978944c9-d91e-493f-8769-d6dacb9c679d","Type":"ContainerStarted","Data":"0ec2cdc63d0a517ab2077b6be9e6f3a01352d3246dd2c4546d3ea9ce6d6b26c1"}
Nov 03 22:02:41.643416 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:41.643388    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb" event={"ID":"cb5cb186-a9c9-491b-8798-77e9cb223d27","Type":"ContainerStarted","Data":"eba93d88f6109ce67d0d8c17d5609671f45dc9ad4679d274e90475c76a2e3c92"}
Nov 03 22:02:41.643559 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:41.643439    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb" event={"ID":"cb5cb186-a9c9-491b-8798-77e9cb223d27","Type":"ContainerStarted","Data":"f86ba2f49835eee7ec41330a2d4542c95029c5240f713603d15aafe02a57755c"}
Nov 03 22:02:41.759545 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:41.759487    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2" podStartSLOduration=2.759468894 podStartE2EDuration="2.759468894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:41.756454587 +0000 UTC m=+2876.250376462" watchObservedRunningTime="2026-11-03 22:02:41.759468894 +0000 UTC m=+2876.253390770"
Nov 03 22:02:41.888311 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:41.887574    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb" podStartSLOduration=2.887557073 podStartE2EDuration="2.887557073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:41.881079052 +0000 UTC m=+2876.375000928" watchObservedRunningTime="2026-11-03 22:02:41.887557073 +0000 UTC m=+2876.381479212"
Nov 03 22:02:42.327868 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:42.325667    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:42.333263 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:42.330800    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 03 22:02:42.333263 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:42.330866    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb" podUID="cb5cb186-a9c9-491b-8798-77e9cb223d27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 03 22:02:42.577993 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:42.577858    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:42.579447 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:42.579413    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Nov 03 22:02:42.579596 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:42.579462    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2" podUID="978944c9-d91e-493f-8769-d6dacb9c679d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Nov 03 22:02:43.328000 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:43.327719    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 03 22:02:43.328000 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:43.327782    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb" podUID="cb5cb186-a9c9-491b-8798-77e9cb223d27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 03 22:02:43.577797 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:43.577757    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Nov 03 22:02:43.578141 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:43.578075    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2" podUID="978944c9-d91e-493f-8769-d6dacb9c679d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Nov 03 22:02:44.330266 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:44.329772    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 03 22:02:44.330266 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:44.329841    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb" podUID="cb5cb186-a9c9-491b-8798-77e9cb223d27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 03 22:02:44.577692 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:44.577654    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Nov 03 22:02:44.577881 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:44.577709    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2" podUID="978944c9-d91e-493f-8769-d6dacb9c679d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Nov 03 22:02:45.327684 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:45.327645    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 03 22:02:45.327991 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:45.327957    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb" podUID="cb5cb186-a9c9-491b-8798-77e9cb223d27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 03 22:02:45.578564 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:45.578485    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Nov 03 22:02:45.579026 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:45.579000    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2" podUID="978944c9-d91e-493f-8769-d6dacb9c679d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Nov 03 22:02:46.329263 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.326683    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 03 22:02:46.329263 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.326747    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb" podUID="cb5cb186-a9c9-491b-8798-77e9cb223d27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 03 22:02:46.479312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.478295    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"]
Nov 03 22:02:46.485110 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.485070    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.577841 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.577803    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Nov 03 22:02:46.578064 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.577869    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2" podUID="978944c9-d91e-493f-8769-d6dacb9c679d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Nov 03 22:02:46.680363 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.651468    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkbd7\" (UniqueName: \"kubernetes.io/projected/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-kube-api-access-pkbd7\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.680363 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.651551    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.680363 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.651588    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.680363 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.651620    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.680363 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.651773    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.680363 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.651870    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.680363 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.651900    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.680363 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.651962    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.680363 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.652001    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.683578 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.680699    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"]
Nov 03 22:02:46.773679 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.773627    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkbd7\" (UniqueName: \"kubernetes.io/projected/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-kube-api-access-pkbd7\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.773904 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.773739    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.773904 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.773778    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.773904 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.773810    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.774095 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.774010    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.774230 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.774127    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.774230 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.774159    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.774230 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.774230    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.774230 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.774329    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.776707 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.776672    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.776832 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.776704    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.777006 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.776977    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.777083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.776992    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.777504 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.777463    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.779226 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.778788    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.780164 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.779916    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.783829 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.783803    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1324f3a4-lz8vw"]
Nov 03 22:02:46.789478 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.789458    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1324f3a4-lz8vw"
Nov 03 22:02:46.865829 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.865701    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkbd7\" (UniqueName: \"kubernetes.io/projected/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-kube-api-access-pkbd7\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.866405 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.866036    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:02:46.866405 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.866103    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:02:46.889536 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.889483    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1324f3a4-lz8vw"]
Nov 03 22:02:46.890657 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.889768    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a601e88-a602-4478-a0c7-2d1e07852968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1324f3a4-lz8vw\" (UID: \"0a601e88-a602-4478-a0c7-2d1e07852968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1324f3a4-lz8vw"
Nov 03 22:02:46.890657 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.889957    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdh8t\" (UniqueName: \"kubernetes.io/projected/0a601e88-a602-4478-a0c7-2d1e07852968-kube-api-access-kdh8t\") pod \"maas-api-revoke-sub-1324f3a4-lz8vw\" (UID: \"0a601e88-a602-4478-a0c7-2d1e07852968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1324f3a4-lz8vw"
Nov 03 22:02:46.890657 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.890026    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a601e88-a602-4478-a0c7-2d1e07852968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1324f3a4-lz8vw\" (UID: \"0a601e88-a602-4478-a0c7-2d1e07852968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1324f3a4-lz8vw"
Nov 03 22:02:46.890657 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.890565    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:46.990958 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.990929    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdh8t\" (UniqueName: \"kubernetes.io/projected/0a601e88-a602-4478-a0c7-2d1e07852968-kube-api-access-kdh8t\") pod \"maas-api-revoke-sub-1324f3a4-lz8vw\" (UID: \"0a601e88-a602-4478-a0c7-2d1e07852968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1324f3a4-lz8vw"
Nov 03 22:02:46.991111 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.990994    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a601e88-a602-4478-a0c7-2d1e07852968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1324f3a4-lz8vw\" (UID: \"0a601e88-a602-4478-a0c7-2d1e07852968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1324f3a4-lz8vw"
Nov 03 22:02:46.991111 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.991093    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a601e88-a602-4478-a0c7-2d1e07852968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1324f3a4-lz8vw\" (UID: \"0a601e88-a602-4478-a0c7-2d1e07852968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1324f3a4-lz8vw"
Nov 03 22:02:46.991820 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.991790    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a601e88-a602-4478-a0c7-2d1e07852968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1324f3a4-lz8vw\" (UID: \"0a601e88-a602-4478-a0c7-2d1e07852968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1324f3a4-lz8vw"
Nov 03 22:02:46.994060 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:46.994027    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a601e88-a602-4478-a0c7-2d1e07852968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1324f3a4-lz8vw\" (UID: \"0a601e88-a602-4478-a0c7-2d1e07852968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1324f3a4-lz8vw"
Nov 03 22:02:47.075121 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:47.074457    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdh8t\" (UniqueName: \"kubernetes.io/projected/0a601e88-a602-4478-a0c7-2d1e07852968-kube-api-access-kdh8t\") pod \"maas-api-revoke-sub-1324f3a4-lz8vw\" (UID: \"0a601e88-a602-4478-a0c7-2d1e07852968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1324f3a4-lz8vw"
Nov 03 22:02:47.144263 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:47.143520    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:47.294087 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:47.293998    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1324f3a4-lz8vw"
Nov 03 22:02:47.332324 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:47.331807    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 03 22:02:47.332324 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:47.331863    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb" podUID="cb5cb186-a9c9-491b-8798-77e9cb223d27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 03 22:02:47.577990 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:47.577896    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Nov 03 22:02:47.577990 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:47.577968    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2" podUID="978944c9-d91e-493f-8769-d6dacb9c679d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Nov 03 22:02:47.642195 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:47.642166    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"]
Nov 03 22:02:47.646370 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:02:47.646334    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb7d89c_af78_4e10_bfe9_ccbbb51e9222.slice/crio-fab0d0299576b6d9b4e0a31d26961e5d5505b32cce12a49c57ffd1ac9a76cae8 WatchSource:0}: Error finding container fab0d0299576b6d9b4e0a31d26961e5d5505b32cce12a49c57ffd1ac9a76cae8: Status 404 returned error can't find the container with id fab0d0299576b6d9b4e0a31d26961e5d5505b32cce12a49c57ffd1ac9a76cae8
Nov 03 22:02:47.654531 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:47.651312    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:02:47.654531 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:47.651406    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:02:47.654531 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:47.651454    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:02:47.780008 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:47.779957    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1324f3a4-lz8vw"]
Nov 03 22:02:47.780497 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:02:47.780461    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a601e88_a602_4478_a0c7_2d1e07852968.slice/crio-495c9ef54d62bbdaad89e8132812be1908e4a3c3907d71a1b62ce64da8e2c63a WatchSource:0}: Error finding container 495c9ef54d62bbdaad89e8132812be1908e4a3c3907d71a1b62ce64da8e2c63a: Status 404 returned error can't find the container with id 495c9ef54d62bbdaad89e8132812be1908e4a3c3907d71a1b62ce64da8e2c63a
Nov 03 22:02:47.995271 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:47.994661    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1324f3a4-lz8vw" event={"ID":"0a601e88-a602-4478-a0c7-2d1e07852968","Type":"ContainerStarted","Data":"495c9ef54d62bbdaad89e8132812be1908e4a3c3907d71a1b62ce64da8e2c63a"}
Nov 03 22:02:47.996759 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:47.996724    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx" event={"ID":"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222","Type":"ContainerStarted","Data":"46a20bb03cfa578f8967ea82fcf7b0154b772d24ce057acac6e3ac9f4d8a60ea"}
Nov 03 22:02:47.996934 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:47.996779    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx" event={"ID":"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222","Type":"ContainerStarted","Data":"fab0d0299576b6d9b4e0a31d26961e5d5505b32cce12a49c57ffd1ac9a76cae8"}
Nov 03 22:02:48.121149 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:48.121016    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx" podStartSLOduration=2.12099723 podStartE2EDuration="2.12099723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:48.115603821 +0000 UTC m=+2882.609525701" watchObservedRunningTime="2026-11-03 22:02:48.12099723 +0000 UTC m=+2882.614919108"
Nov 03 22:02:48.145955 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:48.145917    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:48.146170 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:48.146050    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 22:02:48.146170 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:48.146095    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx" podUID="5fb7d89c-af78-4e10-bfe9-ccbbb51e9222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 22:02:48.580396 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:48.580353    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Nov 03 22:02:48.580628 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:48.580422    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2" podUID="978944c9-d91e-493f-8769-d6dacb9c679d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Nov 03 22:02:48.886009 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:48.885869    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:48.887067 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:48.887042    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:48.888872 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:48.888849    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:02:49.067601 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:49.067527    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1324f3a4-lz8vw" event={"ID":"0a601e88-a602-4478-a0c7-2d1e07852968","Type":"ContainerStarted","Data":"f3bdb9151729a99e3ebf8e5d4ffc1a8b639822dcdfe09e4fd9771fe30e6eecd2"}
Nov 03 22:02:49.146310 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:49.146173    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 22:02:49.146460 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:49.146228    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx" podUID="5fb7d89c-af78-4e10-bfe9-ccbbb51e9222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 22:02:49.281596 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:49.281537    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1324f3a4-lz8vw" podStartSLOduration=3.28151801 podStartE2EDuration="3.28151801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:49.277484819 +0000 UTC m=+2883.771406694" watchObservedRunningTime="2026-11-03 22:02:49.28151801 +0000 UTC m=+2883.775439888"
Nov 03 22:02:49.582604 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:49.582574    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:49.583032 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:49.583006    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:49.583853 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:49.583835    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:02:50.076027 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:50.075991    5053 generic.go:356] "Generic (PLEG): container finished" podID="0a601e88-a602-4478-a0c7-2d1e07852968" containerID="f3bdb9151729a99e3ebf8e5d4ffc1a8b639822dcdfe09e4fd9771fe30e6eecd2" exitCode=0
Nov 03 22:02:50.076584 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:50.076067    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1324f3a4-lz8vw" event={"ID":"0a601e88-a602-4478-a0c7-2d1e07852968","Type":"ContainerDied","Data":"f3bdb9151729a99e3ebf8e5d4ffc1a8b639822dcdfe09e4fd9771fe30e6eecd2"}
Nov 03 22:02:50.144949 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:50.144912    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 22:02:50.145165 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:50.144978    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx" podUID="5fb7d89c-af78-4e10-bfe9-ccbbb51e9222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 22:02:51.144409 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:51.144360    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 22:02:51.144894 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:51.144435    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx" podUID="5fb7d89c-af78-4e10-bfe9-ccbbb51e9222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 22:02:51.354766 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:51.354683    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1324f3a4-lz8vw"
Nov 03 22:02:51.438606 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:51.438566    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a601e88-a602-4478-a0c7-2d1e07852968-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a601e88-a602-4478-a0c7-2d1e07852968-maas-api-xks-ca\") pod \"0a601e88-a602-4478-a0c7-2d1e07852968\" (UID: \"0a601e88-a602-4478-a0c7-2d1e07852968\") "
Nov 03 22:02:51.438815 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:51.438664    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a601e88-a602-4478-a0c7-2d1e07852968-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a601e88-a602-4478-a0c7-2d1e07852968-maas-api-openshift-ca\") pod \"0a601e88-a602-4478-a0c7-2d1e07852968\" (UID: \"0a601e88-a602-4478-a0c7-2d1e07852968\") "
Nov 03 22:02:51.438815 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:51.438688    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a601e88-a602-4478-a0c7-2d1e07852968-kube-api-access-kdh8t\" (UniqueName: \"kubernetes.io/projected/0a601e88-a602-4478-a0c7-2d1e07852968-kube-api-access-kdh8t\") pod \"0a601e88-a602-4478-a0c7-2d1e07852968\" (UID: \"0a601e88-a602-4478-a0c7-2d1e07852968\") "
Nov 03 22:02:51.439130 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:51.439097    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a601e88-a602-4478-a0c7-2d1e07852968-maas-api-openshift-ca" pod "0a601e88-a602-4478-a0c7-2d1e07852968" (UID: "0a601e88-a602-4478-a0c7-2d1e07852968"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:02:51.439407 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:51.439379    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a601e88-a602-4478-a0c7-2d1e07852968-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:51.441248 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:51.441206    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a601e88-a602-4478-a0c7-2d1e07852968-kube-api-access-kdh8t" pod "0a601e88-a602-4478-a0c7-2d1e07852968" (UID: "0a601e88-a602-4478-a0c7-2d1e07852968"). InnerVolumeSpecName "kube-api-access-kdh8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:02:51.441439 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:51.441418    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a601e88-a602-4478-a0c7-2d1e07852968-maas-api-xks-ca" pod "0a601e88-a602-4478-a0c7-2d1e07852968" (UID: "0a601e88-a602-4478-a0c7-2d1e07852968"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:02:51.546266 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:51.545378    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a601e88-a602-4478-a0c7-2d1e07852968-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:51.556407 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:51.556374    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdh8t\" (UniqueName: \"kubernetes.io/projected/0a601e88-a602-4478-a0c7-2d1e07852968-kube-api-access-kdh8t\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:52.092589 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:52.092560    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1324f3a4-lz8vw"
Nov 03 22:02:52.092810 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:52.092556    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1324f3a4-lz8vw" event={"ID":"0a601e88-a602-4478-a0c7-2d1e07852968","Type":"ContainerDied","Data":"495c9ef54d62bbdaad89e8132812be1908e4a3c3907d71a1b62ce64da8e2c63a"}
Nov 03 22:02:52.092810 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:52.092690    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="495c9ef54d62bbdaad89e8132812be1908e4a3c3907d71a1b62ce64da8e2c63a"
Nov 03 22:02:52.144410 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:52.144375    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 22:02:52.144605 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:52.144437    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx" podUID="5fb7d89c-af78-4e10-bfe9-ccbbb51e9222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 22:02:53.144845 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:53.144796    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 22:02:53.145294 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:53.144901    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx" podUID="5fb7d89c-af78-4e10-bfe9-ccbbb51e9222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 22:02:54.144450 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:54.144416    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 22:02:54.144655 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:54.144476    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx" podUID="5fb7d89c-af78-4e10-bfe9-ccbbb51e9222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 22:02:55.144411 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:55.144363    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 22:02:55.144967 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:55.144440    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx" podUID="5fb7d89c-af78-4e10-bfe9-ccbbb51e9222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 22:02:57.229264 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:57.219379    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:02:57.229264 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:57.219488    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx" podUID="5fb7d89c-af78-4e10-bfe9-ccbbb51e9222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:02:57.584347 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:57.584254    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:57.584692 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:57.584671    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:02:57.585459 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:02:57.585440    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:03:10.929290 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:03:10.929211    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:03:31.411686 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:31.411597    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"]
Nov 03 22:03:31.412282 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:31.411882    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6" podUID="604c02df-5b27-4ed4-9e65-a32f6f7207d0" containerName="maas-api" containerID="cri-o://ddc0cacf5da1f00985937feb3c265797c8ac2031152aa27fa2eb6534c7bd5256" gracePeriod=30
Nov 03 22:03:31.589412 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:31.589368    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6" podUID="604c02df-5b27-4ed4-9e65-a32f6f7207d0" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.203:8443/health\": dial tcp 10.132.0.203:8443: connect: connection refused"
Nov 03 22:03:31.739685 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:31.739654    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"
Nov 03 22:03:31.875970 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:31.875936    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604c02df-5b27-4ed4-9e65-a32f6f7207d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/604c02df-5b27-4ed4-9e65-a32f6f7207d0-metrics-certs\") pod \"604c02df-5b27-4ed4-9e65-a32f6f7207d0\" (UID: \"604c02df-5b27-4ed4-9e65-a32f6f7207d0\") "
Nov 03 22:03:31.876154 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:31.876138    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604c02df-5b27-4ed4-9e65-a32f6f7207d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604c02df-5b27-4ed4-9e65-a32f6f7207d0-maas-api-tls\") pod \"604c02df-5b27-4ed4-9e65-a32f6f7207d0\" (UID: \"604c02df-5b27-4ed4-9e65-a32f6f7207d0\") "
Nov 03 22:03:31.876253 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:31.876173    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604c02df-5b27-4ed4-9e65-a32f6f7207d0-kube-api-access-nr8r5\" (UniqueName: \"kubernetes.io/projected/604c02df-5b27-4ed4-9e65-a32f6f7207d0-kube-api-access-nr8r5\") pod \"604c02df-5b27-4ed4-9e65-a32f6f7207d0\" (UID: \"604c02df-5b27-4ed4-9e65-a32f6f7207d0\") "
Nov 03 22:03:31.876324 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:31.876299    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604c02df-5b27-4ed4-9e65-a32f6f7207d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604c02df-5b27-4ed4-9e65-a32f6f7207d0-openshift-service-ca\") pod \"604c02df-5b27-4ed4-9e65-a32f6f7207d0\" (UID: \"604c02df-5b27-4ed4-9e65-a32f6f7207d0\") "
Nov 03 22:03:31.877171 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:31.877145    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604c02df-5b27-4ed4-9e65-a32f6f7207d0-openshift-service-ca" pod "604c02df-5b27-4ed4-9e65-a32f6f7207d0" (UID: "604c02df-5b27-4ed4-9e65-a32f6f7207d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:03:31.879727 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:31.879698    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604c02df-5b27-4ed4-9e65-a32f6f7207d0-kube-api-access-nr8r5" pod "604c02df-5b27-4ed4-9e65-a32f6f7207d0" (UID: "604c02df-5b27-4ed4-9e65-a32f6f7207d0"). InnerVolumeSpecName "kube-api-access-nr8r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:03:31.880623 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:31.880552    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604c02df-5b27-4ed4-9e65-a32f6f7207d0-metrics-certs" pod "604c02df-5b27-4ed4-9e65-a32f6f7207d0" (UID: "604c02df-5b27-4ed4-9e65-a32f6f7207d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:03:31.882472 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:31.882196    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604c02df-5b27-4ed4-9e65-a32f6f7207d0-maas-api-tls" pod "604c02df-5b27-4ed4-9e65-a32f6f7207d0" (UID: "604c02df-5b27-4ed4-9e65-a32f6f7207d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:03:31.995899 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:31.994369    5053 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604c02df-5b27-4ed4-9e65-a32f6f7207d0-openshift-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:31.995899 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:31.994482    5053 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/604c02df-5b27-4ed4-9e65-a32f6f7207d0-metrics-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:31.995899 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:31.994499    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604c02df-5b27-4ed4-9e65-a32f6f7207d0-maas-api-tls\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:31.995899 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:31.994509    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr8r5\" (UniqueName: \"kubernetes.io/projected/604c02df-5b27-4ed4-9e65-a32f6f7207d0-kube-api-access-nr8r5\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:32.524220 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:32.524185    5053 generic.go:356] "Generic (PLEG): container finished" podID="604c02df-5b27-4ed4-9e65-a32f6f7207d0" containerID="ddc0cacf5da1f00985937feb3c265797c8ac2031152aa27fa2eb6534c7bd5256" exitCode=0
Nov 03 22:03:32.524714 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:32.524251    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"
Nov 03 22:03:32.524714 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:32.524269    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6" event={"ID":"604c02df-5b27-4ed4-9e65-a32f6f7207d0","Type":"ContainerDied","Data":"ddc0cacf5da1f00985937feb3c265797c8ac2031152aa27fa2eb6534c7bd5256"}
Nov 03 22:03:32.524714 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:32.524332    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6" event={"ID":"604c02df-5b27-4ed4-9e65-a32f6f7207d0","Type":"ContainerDied","Data":"09e6f1efa51d377a0408c09afe38ca5eb09bffe192ccb0811ce5c07cbc22f5d8"}
Nov 03 22:03:32.524714 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:32.524362    5053 scope.go:122] "RemoveContainer" containerID="ddc0cacf5da1f00985937feb3c265797c8ac2031152aa27fa2eb6534c7bd5256"
Nov 03 22:03:32.537668 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:32.537644    5053 scope.go:122] "RemoveContainer" containerID="ddc0cacf5da1f00985937feb3c265797c8ac2031152aa27fa2eb6534c7bd5256"
Nov 03 22:03:32.538030 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:03:32.538007    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddc0cacf5da1f00985937feb3c265797c8ac2031152aa27fa2eb6534c7bd5256\": container with ID starting with ddc0cacf5da1f00985937feb3c265797c8ac2031152aa27fa2eb6534c7bd5256 not found: ID does not exist" containerID="ddc0cacf5da1f00985937feb3c265797c8ac2031152aa27fa2eb6534c7bd5256"
Nov 03 22:03:32.538109 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:32.538043    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddc0cacf5da1f00985937feb3c265797c8ac2031152aa27fa2eb6534c7bd5256"} err="failed to get container status \"ddc0cacf5da1f00985937feb3c265797c8ac2031152aa27fa2eb6534c7bd5256\": rpc error: code = NotFound desc = could not find container \"ddc0cacf5da1f00985937feb3c265797c8ac2031152aa27fa2eb6534c7bd5256\": container with ID starting with ddc0cacf5da1f00985937feb3c265797c8ac2031152aa27fa2eb6534c7bd5256 not found: ID does not exist"
Nov 03 22:03:32.688580 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:32.688548    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"]
Nov 03 22:03:32.738556 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:32.738517    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4644ae-876fd5765-stdp6"]
Nov 03 22:03:33.669570 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:33.669536    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="604c02df-5b27-4ed4-9e65-a32f6f7207d0" path="/var/lib/kubelet/pods/604c02df-5b27-4ed4-9e65-a32f6f7207d0/volumes"
Nov 03 22:03:40.193418 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:40.193379    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"]
Nov 03 22:03:40.193971 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:40.193703    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk" podUID="720e4837-f945-4df0-a637-c2dc41608b7e" containerName="istio-proxy" containerID="cri-o://90592ab9d76ada256f68841bce1fb4c6b331c10080d57e0fe48b853674cb4fc9" gracePeriod=30
Nov 03 22:03:40.931668 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:03:40.931631    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:03:42.340813 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:42.340776    5053 patch_prober.go:29] interesting pod/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:03:42.341397 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:42.340854    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk" podUID="720e4837-f945-4df0-a637-c2dc41608b7e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:03:45.493760 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.493733    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 22:03:45.605756 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.605716    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-credential-socket\") pod \"720e4837-f945-4df0-a637-c2dc41608b7e\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") "
Nov 03 22:03:45.605976 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.605796    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-workload-certs\") pod \"720e4837-f945-4df0-a637-c2dc41608b7e\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") "
Nov 03 22:03:45.605976 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.605862    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/720e4837-f945-4df0-a637-c2dc41608b7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720e4837-f945-4df0-a637-c2dc41608b7e-istio-podinfo\") pod \"720e4837-f945-4df0-a637-c2dc41608b7e\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") "
Nov 03 22:03:45.605976 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.605925    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-istio-envoy\") pod \"720e4837-f945-4df0-a637-c2dc41608b7e\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") "
Nov 03 22:03:45.606174 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.606013    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/720e4837-f945-4df0-a637-c2dc41608b7e-istio-token\" (UniqueName: \"kubernetes.io/projected/720e4837-f945-4df0-a637-c2dc41608b7e-istio-token\") pod \"720e4837-f945-4df0-a637-c2dc41608b7e\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") "
Nov 03 22:03:45.606174 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.606092    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-credential-socket" pod "720e4837-f945-4df0-a637-c2dc41608b7e" (UID: "720e4837-f945-4df0-a637-c2dc41608b7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:45.606174 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.606149    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-workload-socket\") pod \"720e4837-f945-4df0-a637-c2dc41608b7e\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") "
Nov 03 22:03:45.606381 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.606121    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-workload-certs" pod "720e4837-f945-4df0-a637-c2dc41608b7e" (UID: "720e4837-f945-4df0-a637-c2dc41608b7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:45.606381 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.606317    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/720e4837-f945-4df0-a637-c2dc41608b7e-kube-api-access-tv9nw\" (UniqueName: \"kubernetes.io/projected/720e4837-f945-4df0-a637-c2dc41608b7e-kube-api-access-tv9nw\") pod \"720e4837-f945-4df0-a637-c2dc41608b7e\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") "
Nov 03 22:03:45.606491 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.606409    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-istio-data\") pod \"720e4837-f945-4df0-a637-c2dc41608b7e\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") "
Nov 03 22:03:45.606491 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.606413    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-workload-socket" pod "720e4837-f945-4df0-a637-c2dc41608b7e" (UID: "720e4837-f945-4df0-a637-c2dc41608b7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:45.606491 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.606456    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/720e4837-f945-4df0-a637-c2dc41608b7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720e4837-f945-4df0-a637-c2dc41608b7e-istiod-ca-cert\") pod \"720e4837-f945-4df0-a637-c2dc41608b7e\" (UID: \"720e4837-f945-4df0-a637-c2dc41608b7e\") "
Nov 03 22:03:45.606683 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.606658    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-istio-data" pod "720e4837-f945-4df0-a637-c2dc41608b7e" (UID: "720e4837-f945-4df0-a637-c2dc41608b7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:45.607048 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.606942    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/720e4837-f945-4df0-a637-c2dc41608b7e-istiod-ca-cert" pod "720e4837-f945-4df0-a637-c2dc41608b7e" (UID: "720e4837-f945-4df0-a637-c2dc41608b7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:03:45.607164 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.607143    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:45.607227 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.607173    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:45.607227 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.607186    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:45.607227 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.607201    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720e4837-f945-4df0-a637-c2dc41608b7e-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:45.607227 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.607215    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:45.609050 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.609014    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/720e4837-f945-4df0-a637-c2dc41608b7e-kube-api-access-tv9nw" pod "720e4837-f945-4df0-a637-c2dc41608b7e" (UID: "720e4837-f945-4df0-a637-c2dc41608b7e"). InnerVolumeSpecName "kube-api-access-tv9nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:03:45.609211 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.609172    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/720e4837-f945-4df0-a637-c2dc41608b7e-istio-token" pod "720e4837-f945-4df0-a637-c2dc41608b7e" (UID: "720e4837-f945-4df0-a637-c2dc41608b7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:03:45.609211 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.609202    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-istio-envoy" pod "720e4837-f945-4df0-a637-c2dc41608b7e" (UID: "720e4837-f945-4df0-a637-c2dc41608b7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:45.610883 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.610858    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/720e4837-f945-4df0-a637-c2dc41608b7e-istio-podinfo" pod "720e4837-f945-4df0-a637-c2dc41608b7e" (UID: "720e4837-f945-4df0-a637-c2dc41608b7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:03:45.627339 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.627265    5053 generic.go:356] "Generic (PLEG): container finished" podID="720e4837-f945-4df0-a637-c2dc41608b7e" containerID="90592ab9d76ada256f68841bce1fb4c6b331c10080d57e0fe48b853674cb4fc9" exitCode=0
Nov 03 22:03:45.627339 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.627303    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk" event={"ID":"720e4837-f945-4df0-a637-c2dc41608b7e","Type":"ContainerDied","Data":"90592ab9d76ada256f68841bce1fb4c6b331c10080d57e0fe48b853674cb4fc9"}
Nov 03 22:03:45.627518 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.627343    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"
Nov 03 22:03:45.627518 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.627417    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk" event={"ID":"720e4837-f945-4df0-a637-c2dc41608b7e","Type":"ContainerDied","Data":"c4f3351c099e67b848450116eaa62ac911a917fd6ca9b887b538671881b53a21"}
Nov 03 22:03:45.627518 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.627450    5053 scope.go:122] "RemoveContainer" containerID="90592ab9d76ada256f68841bce1fb4c6b331c10080d57e0fe48b853674cb4fc9"
Nov 03 22:03:45.685143 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.651688    5053 scope.go:122] "RemoveContainer" containerID="1f4e9cb903c04d892ac73fdfb0537b6103928248909ed4aa97b5c97b7367f471"
Nov 03 22:03:45.700023 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.699894    5053 scope.go:122] "RemoveContainer" containerID="90592ab9d76ada256f68841bce1fb4c6b331c10080d57e0fe48b853674cb4fc9"
Nov 03 22:03:45.700368 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:03:45.700329    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90592ab9d76ada256f68841bce1fb4c6b331c10080d57e0fe48b853674cb4fc9\": container with ID starting with 90592ab9d76ada256f68841bce1fb4c6b331c10080d57e0fe48b853674cb4fc9 not found: ID does not exist" containerID="90592ab9d76ada256f68841bce1fb4c6b331c10080d57e0fe48b853674cb4fc9"
Nov 03 22:03:45.700477 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.700377    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90592ab9d76ada256f68841bce1fb4c6b331c10080d57e0fe48b853674cb4fc9"} err="failed to get container status \"90592ab9d76ada256f68841bce1fb4c6b331c10080d57e0fe48b853674cb4fc9\": rpc error: code = NotFound desc = could not find container \"90592ab9d76ada256f68841bce1fb4c6b331c10080d57e0fe48b853674cb4fc9\": container with ID starting with 90592ab9d76ada256f68841bce1fb4c6b331c10080d57e0fe48b853674cb4fc9 not found: ID does not exist"
Nov 03 22:03:45.700477 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.700404    5053 scope.go:122] "RemoveContainer" containerID="1f4e9cb903c04d892ac73fdfb0537b6103928248909ed4aa97b5c97b7367f471"
Nov 03 22:03:45.700750 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:03:45.700723    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f4e9cb903c04d892ac73fdfb0537b6103928248909ed4aa97b5c97b7367f471\": container with ID starting with 1f4e9cb903c04d892ac73fdfb0537b6103928248909ed4aa97b5c97b7367f471 not found: ID does not exist" containerID="1f4e9cb903c04d892ac73fdfb0537b6103928248909ed4aa97b5c97b7367f471"
Nov 03 22:03:45.700809 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.700758    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4e9cb903c04d892ac73fdfb0537b6103928248909ed4aa97b5c97b7367f471"} err="failed to get container status \"1f4e9cb903c04d892ac73fdfb0537b6103928248909ed4aa97b5c97b7367f471\": rpc error: code = NotFound desc = could not find container \"1f4e9cb903c04d892ac73fdfb0537b6103928248909ed4aa97b5c97b7367f471\": container with ID starting with 1f4e9cb903c04d892ac73fdfb0537b6103928248909ed4aa97b5c97b7367f471 not found: ID does not exist"
Nov 03 22:03:45.708954 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.708924    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720e4837-f945-4df0-a637-c2dc41608b7e-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:45.709068 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.708960    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720e4837-f945-4df0-a637-c2dc41608b7e-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:45.709068 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.708975    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720e4837-f945-4df0-a637-c2dc41608b7e-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:45.709068 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.708989    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv9nw\" (UniqueName: \"kubernetes.io/projected/720e4837-f945-4df0-a637-c2dc41608b7e-kube-api-access-tv9nw\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:45.848610 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.848517    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"]
Nov 03 22:03:45.853340 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:45.853309    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-4644ae-openshift-default-9d89fcbc7-xknmk"]
Nov 03 22:03:46.241994 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.241959    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n"]
Nov 03 22:03:46.242612 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.242598    5053 cpu_manager.go:414] "containerMap: removing container" podUID="0a601e88-a602-4478-a0c7-2d1e07852968" containerName="revoke-keys"
Nov 03 22:03:46.242669 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.242613    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a601e88-a602-4478-a0c7-2d1e07852968" containerName="revoke-keys"
Nov 03 22:03:46.242669 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.242635    5053 cpu_manager.go:414] "containerMap: removing container" podUID="720e4837-f945-4df0-a637-c2dc41608b7e" containerName="istio-proxy"
Nov 03 22:03:46.242669 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.242640    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="720e4837-f945-4df0-a637-c2dc41608b7e" containerName="istio-proxy"
Nov 03 22:03:46.242669 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.242664    5053 cpu_manager.go:414] "containerMap: removing container" podUID="720e4837-f945-4df0-a637-c2dc41608b7e" containerName="istio-proxy"
Nov 03 22:03:46.242669 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.242669    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="720e4837-f945-4df0-a637-c2dc41608b7e" containerName="istio-proxy"
Nov 03 22:03:46.242827 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.242697    5053 cpu_manager.go:414] "containerMap: removing container" podUID="604c02df-5b27-4ed4-9e65-a32f6f7207d0" containerName="maas-api"
Nov 03 22:03:46.242827 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.242702    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604c02df-5b27-4ed4-9e65-a32f6f7207d0" containerName="maas-api"
Nov 03 22:03:46.242827 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.242789    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="720e4837-f945-4df0-a637-c2dc41608b7e" containerName="istio-proxy"
Nov 03 22:03:46.242827 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.242803    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="604c02df-5b27-4ed4-9e65-a32f6f7207d0" containerName="maas-api"
Nov 03 22:03:46.242827 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.242813    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a601e88-a602-4478-a0c7-2d1e07852968" containerName="revoke-keys"
Nov 03 22:03:46.247777 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.247757    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n"
Nov 03 22:03:46.300086 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.300055    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:03:46.300295 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.300186    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:03:46.303441 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.303410    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n"]
Nov 03 22:03:46.329153 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.329118    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4030fc07-5675-4b89-bb00-4f026cebb1b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n\" (UID: \"4030fc07-5675-4b89-bb00-4f026cebb1b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n"
Nov 03 22:03:46.329389 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.329365    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h52k\" (UniqueName: \"kubernetes.io/projected/4030fc07-5675-4b89-bb00-4f026cebb1b4-kube-api-access-4h52k\") pod \"maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n\" (UID: \"4030fc07-5675-4b89-bb00-4f026cebb1b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n"
Nov 03 22:03:46.329482 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.329466    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4030fc07-5675-4b89-bb00-4f026cebb1b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n\" (UID: \"4030fc07-5675-4b89-bb00-4f026cebb1b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n"
Nov 03 22:03:46.430797 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.430757    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4030fc07-5675-4b89-bb00-4f026cebb1b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n\" (UID: \"4030fc07-5675-4b89-bb00-4f026cebb1b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n"
Nov 03 22:03:46.430993 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.430877    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h52k\" (UniqueName: \"kubernetes.io/projected/4030fc07-5675-4b89-bb00-4f026cebb1b4-kube-api-access-4h52k\") pod \"maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n\" (UID: \"4030fc07-5675-4b89-bb00-4f026cebb1b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n"
Nov 03 22:03:46.430993 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.430936    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4030fc07-5675-4b89-bb00-4f026cebb1b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n\" (UID: \"4030fc07-5675-4b89-bb00-4f026cebb1b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n"
Nov 03 22:03:46.431738 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.431662    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4030fc07-5675-4b89-bb00-4f026cebb1b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n\" (UID: \"4030fc07-5675-4b89-bb00-4f026cebb1b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n"
Nov 03 22:03:46.433582 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.433556    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4030fc07-5675-4b89-bb00-4f026cebb1b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n\" (UID: \"4030fc07-5675-4b89-bb00-4f026cebb1b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n"
Nov 03 22:03:46.537574 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.537491    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h52k\" (UniqueName: \"kubernetes.io/projected/4030fc07-5675-4b89-bb00-4f026cebb1b4-kube-api-access-4h52k\") pod \"maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n\" (UID: \"4030fc07-5675-4b89-bb00-4f026cebb1b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n"
Nov 03 22:03:46.562260 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.562204    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n"
Nov 03 22:03:46.948411 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:03:46.948373    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4030fc07_5675_4b89_bb00_4f026cebb1b4.slice/crio-b6e7fa0e5d6cda6e0b4c86a75c46e8987e3afe7299bab4139c7aea72ce0fe3dd WatchSource:0}: Error finding container b6e7fa0e5d6cda6e0b4c86a75c46e8987e3afe7299bab4139c7aea72ce0fe3dd: Status 404 returned error can't find the container with id b6e7fa0e5d6cda6e0b4c86a75c46e8987e3afe7299bab4139c7aea72ce0fe3dd
Nov 03 22:03:46.948543 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:46.948436    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n"]
Nov 03 22:03:47.674342 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:47.674230    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="720e4837-f945-4df0-a637-c2dc41608b7e" path="/var/lib/kubelet/pods/720e4837-f945-4df0-a637-c2dc41608b7e/volumes"
Nov 03 22:03:47.674881 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:47.674594    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n" event={"ID":"4030fc07-5675-4b89-bb00-4f026cebb1b4","Type":"ContainerStarted","Data":"3044d098994c8a9d2da7d365972dd5f3c8613ddf44c6588f4707ce4fee3cddc2"}
Nov 03 22:03:47.674881 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:47.674662    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n" event={"ID":"4030fc07-5675-4b89-bb00-4f026cebb1b4","Type":"ContainerStarted","Data":"b6e7fa0e5d6cda6e0b4c86a75c46e8987e3afe7299bab4139c7aea72ce0fe3dd"}
Nov 03 22:03:47.719993 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:47.719967    5053 scope.go:122] "RemoveContainer" containerID="7997c4f2b762321498668018d0fc3b6ae73ea1cfc5cb889c8dcd787c65c745c3"
Nov 03 22:03:47.731051 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:47.731017    5053 scope.go:122] "RemoveContainer" containerID="d9bf3f37c02f73226a5d3b8aee910e07feb8d84686929e8b7eb8dc1db4ecc403"
Nov 03 22:03:47.838186 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:47.838123    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n" podStartSLOduration=1.838105082 podStartE2EDuration="1.838105082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:03:47.832841542 +0000 UTC m=+2942.326763419" watchObservedRunningTime="2026-11-03 22:03:47.838105082 +0000 UTC m=+2942.332026960"
Nov 03 22:03:48.685877 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:48.685836    5053 generic.go:356] "Generic (PLEG): container finished" podID="4030fc07-5675-4b89-bb00-4f026cebb1b4" containerID="3044d098994c8a9d2da7d365972dd5f3c8613ddf44c6588f4707ce4fee3cddc2" exitCode=0
Nov 03 22:03:48.686494 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:48.685948    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n" event={"ID":"4030fc07-5675-4b89-bb00-4f026cebb1b4","Type":"ContainerDied","Data":"3044d098994c8a9d2da7d365972dd5f3c8613ddf44c6588f4707ce4fee3cddc2"}
Nov 03 22:03:49.994449 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:49.994419    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n"
Nov 03 22:03:50.018708 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:50.018666    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4030fc07-5675-4b89-bb00-4f026cebb1b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4030fc07-5675-4b89-bb00-4f026cebb1b4-maas-api-openshift-ca\") pod \"4030fc07-5675-4b89-bb00-4f026cebb1b4\" (UID: \"4030fc07-5675-4b89-bb00-4f026cebb1b4\") "
Nov 03 22:03:50.018708 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:50.018709    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4030fc07-5675-4b89-bb00-4f026cebb1b4-kube-api-access-4h52k\" (UniqueName: \"kubernetes.io/projected/4030fc07-5675-4b89-bb00-4f026cebb1b4-kube-api-access-4h52k\") pod \"4030fc07-5675-4b89-bb00-4f026cebb1b4\" (UID: \"4030fc07-5675-4b89-bb00-4f026cebb1b4\") "
Nov 03 22:03:50.018963 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:50.018775    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4030fc07-5675-4b89-bb00-4f026cebb1b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4030fc07-5675-4b89-bb00-4f026cebb1b4-maas-api-xks-ca\") pod \"4030fc07-5675-4b89-bb00-4f026cebb1b4\" (UID: \"4030fc07-5675-4b89-bb00-4f026cebb1b4\") "
Nov 03 22:03:50.019146 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:50.019119    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4030fc07-5675-4b89-bb00-4f026cebb1b4-maas-api-openshift-ca" pod "4030fc07-5675-4b89-bb00-4f026cebb1b4" (UID: "4030fc07-5675-4b89-bb00-4f026cebb1b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:03:50.019386 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:50.019367    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4030fc07-5675-4b89-bb00-4f026cebb1b4-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:50.021680 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:50.021651    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4030fc07-5675-4b89-bb00-4f026cebb1b4-maas-api-xks-ca" pod "4030fc07-5675-4b89-bb00-4f026cebb1b4" (UID: "4030fc07-5675-4b89-bb00-4f026cebb1b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:03:50.021859 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:50.021795    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4030fc07-5675-4b89-bb00-4f026cebb1b4-kube-api-access-4h52k" pod "4030fc07-5675-4b89-bb00-4f026cebb1b4" (UID: "4030fc07-5675-4b89-bb00-4f026cebb1b4"). InnerVolumeSpecName "kube-api-access-4h52k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:03:50.120984 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:50.120942    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h52k\" (UniqueName: \"kubernetes.io/projected/4030fc07-5675-4b89-bb00-4f026cebb1b4-kube-api-access-4h52k\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:50.120984 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:50.120986    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4030fc07-5675-4b89-bb00-4f026cebb1b4-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:50.738054 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:50.738015    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n"
Nov 03 22:03:50.738054 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:50.738036    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n" event={"ID":"4030fc07-5675-4b89-bb00-4f026cebb1b4","Type":"ContainerDied","Data":"b6e7fa0e5d6cda6e0b4c86a75c46e8987e3afe7299bab4139c7aea72ce0fe3dd"}
Nov 03 22:03:50.738367 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:50.738093    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6e7fa0e5d6cda6e0b4c86a75c46e8987e3afe7299bab4139c7aea72ce0fe3dd"
Nov 03 22:03:57.745027 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:57.744994    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"]
Nov 03 22:03:57.745506 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:03:57.745216    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g" podUID="b8574f16-35c1-4717-a83a-396210ca7a1f" containerName="istio-proxy" containerID="cri-o://66a8879dd6a4aa2f725f314ebc129836d6ebd079e40ea6319901894390e4ff69" gracePeriod=30
Nov 03 22:04:00.394091 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:00.394038    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"]
Nov 03 22:04:00.394928 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:00.394906    5053 cpu_manager.go:414] "containerMap: removing container" podUID="4030fc07-5675-4b89-bb00-4f026cebb1b4" containerName="revoke-keys"
Nov 03 22:04:00.394928 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:00.394929    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4030fc07-5675-4b89-bb00-4f026cebb1b4" containerName="revoke-keys"
Nov 03 22:04:00.395090 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:00.395031    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="720e4837-f945-4df0-a637-c2dc41608b7e" containerName="istio-proxy"
Nov 03 22:04:00.395090 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:00.395042    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="4030fc07-5675-4b89-bb00-4f026cebb1b4" containerName="revoke-keys"
Nov 03 22:04:00.399129 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:00.399110    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"
Nov 03 22:04:00.453243 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:00.453200    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"]
Nov 03 22:04:00.485805 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:00.485770    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2\" (UID: \"5b3a23ab-d256-44d9-9dc1-1b4adff44b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"
Nov 03 22:04:00.486007 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:00.485839    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2\" (UID: \"5b3a23ab-d256-44d9-9dc1-1b4adff44b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"
Nov 03 22:04:00.486007 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:00.485971    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rk76\" (UniqueName: \"kubernetes.io/projected/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-kube-api-access-6rk76\") pod \"maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2\" (UID: \"5b3a23ab-d256-44d9-9dc1-1b4adff44b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"
Nov 03 22:04:00.486118 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:00.486025    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2\" (UID: \"5b3a23ab-d256-44d9-9dc1-1b4adff44b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"
Nov 03 22:04:00.498432 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:00.498393    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-a49a32-serving-cert\""
Nov 03 22:04:00.587468 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:00.587419    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2\" (UID: \"5b3a23ab-d256-44d9-9dc1-1b4adff44b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"
Nov 03 22:04:00.587682 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:00.587508    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rk76\" (UniqueName: \"kubernetes.io/projected/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-kube-api-access-6rk76\") pod \"maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2\" (UID: \"5b3a23ab-d256-44d9-9dc1-1b4adff44b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"
Nov 03 22:04:00.587682 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:04:00.587570    5053 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-serving-cert: secret "maas-api-e2e-shared-a-w0-a49a32-serving-cert" not found
Nov 03 22:04:00.587682 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:04:00.587648    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-maas-api-tls podName:5b3a23ab-d256-44d9-9dc1-1b4adff44b32 nodeName:}" failed. No retries permitted until 2026-11-03 22:04:01.087626873 +0000 UTC m=+2955.581548729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-maas-api-tls") pod "maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2" (UID: "5b3a23ab-d256-44d9-9dc1-1b4adff44b32") : secret "maas-api-e2e-shared-a-w0-a49a32-serving-cert" not found
Nov 03 22:04:00.587870 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:00.587575    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2\" (UID: \"5b3a23ab-d256-44d9-9dc1-1b4adff44b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"
Nov 03 22:04:00.588057 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:00.588028    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2\" (UID: \"5b3a23ab-d256-44d9-9dc1-1b4adff44b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"
Nov 03 22:04:00.588299 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:00.588276    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2\" (UID: \"5b3a23ab-d256-44d9-9dc1-1b4adff44b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"
Nov 03 22:04:00.591321 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:00.591294    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2\" (UID: \"5b3a23ab-d256-44d9-9dc1-1b4adff44b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"
Nov 03 22:04:00.691072 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:00.690987    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rk76\" (UniqueName: \"kubernetes.io/projected/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-kube-api-access-6rk76\") pod \"maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2\" (UID: \"5b3a23ab-d256-44d9-9dc1-1b4adff44b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"
Nov 03 22:04:01.035872 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:01.035818    5053 patch_prober.go:29] interesting pod/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:04:01.036058 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:01.035903    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g" podUID="b8574f16-35c1-4717-a83a-396210ca7a1f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:04:01.095907 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:01.095873    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2\" (UID: \"5b3a23ab-d256-44d9-9dc1-1b4adff44b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"
Nov 03 22:04:01.098726 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:01.098694    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2\" (UID: \"5b3a23ab-d256-44d9-9dc1-1b4adff44b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"
Nov 03 22:04:01.313558 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:01.313465    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"
Nov 03 22:04:01.692387 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:01.692354    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"]
Nov 03 22:04:01.693295 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:04:01.693226    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b3a23ab_d256_44d9_9dc1_1b4adff44b32.slice/crio-d966351927cc67f54ad178a285a5c860dd68ad998575b7266f0f3f89aae58b24 WatchSource:0}: Error finding container d966351927cc67f54ad178a285a5c860dd68ad998575b7266f0f3f89aae58b24: Status 404 returned error can't find the container with id d966351927cc67f54ad178a285a5c860dd68ad998575b7266f0f3f89aae58b24
Nov 03 22:04:01.694643 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:01.694622    5053 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 22:04:01.828227 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:01.828185    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2" event={"ID":"5b3a23ab-d256-44d9-9dc1-1b4adff44b32","Type":"ContainerStarted","Data":"d966351927cc67f54ad178a285a5c860dd68ad998575b7266f0f3f89aae58b24"}
Nov 03 22:04:02.838036 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:02.838004    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2" event={"ID":"5b3a23ab-d256-44d9-9dc1-1b4adff44b32","Type":"ContainerStarted","Data":"b34986ca34b5448f961e3a587464fa6471b6205ab1e2b9b1f4628f5688143da3"}
Nov 03 22:04:02.838499 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:02.838067    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"
Nov 03 22:04:02.959624 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:02.959568    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2" podStartSLOduration=2.124380308 podStartE2EDuration="2.959554041s" totalImagesPullingTime="835.173733ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:04:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:04:02.954261499 +0000 UTC m=+2957.448183375" watchObservedRunningTime="2026-11-03 22:04:02.959554041 +0000 UTC m=+2957.453475915"
Nov 03 22:04:03.049373 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.049281    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 22:04:03.125950 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.125913    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-token\" (UniqueName: \"kubernetes.io/projected/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-token\") pod \"b8574f16-35c1-4717-a83a-396210ca7a1f\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") "
Nov 03 22:04:03.126141 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.125973    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-envoy\") pod \"b8574f16-35c1-4717-a83a-396210ca7a1f\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") "
Nov 03 22:04:03.126141 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.126053    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-podinfo\") pod \"b8574f16-35c1-4717-a83a-396210ca7a1f\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") "
Nov 03 22:04:03.126141 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.126121    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-workload-certs\") pod \"b8574f16-35c1-4717-a83a-396210ca7a1f\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") "
Nov 03 22:04:03.126399 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.126170    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-data\") pod \"b8574f16-35c1-4717-a83a-396210ca7a1f\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") "
Nov 03 22:04:03.126399 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.126218    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-credential-socket\") pod \"b8574f16-35c1-4717-a83a-396210ca7a1f\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") "
Nov 03 22:04:03.126399 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.126264    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8574f16-35c1-4717-a83a-396210ca7a1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8574f16-35c1-4717-a83a-396210ca7a1f-istiod-ca-cert\") pod \"b8574f16-35c1-4717-a83a-396210ca7a1f\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") "
Nov 03 22:04:03.126399 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.126385    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8574f16-35c1-4717-a83a-396210ca7a1f-kube-api-access-8vdjv\" (UniqueName: \"kubernetes.io/projected/b8574f16-35c1-4717-a83a-396210ca7a1f-kube-api-access-8vdjv\") pod \"b8574f16-35c1-4717-a83a-396210ca7a1f\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") "
Nov 03 22:04:03.126615 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.126429    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-workload-socket\") pod \"b8574f16-35c1-4717-a83a-396210ca7a1f\" (UID: \"b8574f16-35c1-4717-a83a-396210ca7a1f\") "
Nov 03 22:04:03.126615 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.126484    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-data" pod "b8574f16-35c1-4717-a83a-396210ca7a1f" (UID: "b8574f16-35c1-4717-a83a-396210ca7a1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:03.126615 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.126576    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-workload-certs" pod "b8574f16-35c1-4717-a83a-396210ca7a1f" (UID: "b8574f16-35c1-4717-a83a-396210ca7a1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:03.126782 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.126685    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-credential-socket" pod "b8574f16-35c1-4717-a83a-396210ca7a1f" (UID: "b8574f16-35c1-4717-a83a-396210ca7a1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:03.126884 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.126849    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-workload-socket" pod "b8574f16-35c1-4717-a83a-396210ca7a1f" (UID: "b8574f16-35c1-4717-a83a-396210ca7a1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:03.126947 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.126897    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8574f16-35c1-4717-a83a-396210ca7a1f-istiod-ca-cert" pod "b8574f16-35c1-4717-a83a-396210ca7a1f" (UID: "b8574f16-35c1-4717-a83a-396210ca7a1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:04:03.127063 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.127045    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:03.127323 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.127068    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:03.127323 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.127082    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:03.127323 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.127095    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8574f16-35c1-4717-a83a-396210ca7a1f-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:03.127323 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.127108    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:03.128971 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.128941    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8574f16-35c1-4717-a83a-396210ca7a1f-kube-api-access-8vdjv" pod "b8574f16-35c1-4717-a83a-396210ca7a1f" (UID: "b8574f16-35c1-4717-a83a-396210ca7a1f"). InnerVolumeSpecName "kube-api-access-8vdjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:04:03.129215 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.129193    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-token" pod "b8574f16-35c1-4717-a83a-396210ca7a1f" (UID: "b8574f16-35c1-4717-a83a-396210ca7a1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:04:03.129439 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.129419    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-podinfo" pod "b8574f16-35c1-4717-a83a-396210ca7a1f" (UID: "b8574f16-35c1-4717-a83a-396210ca7a1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:04:03.129686 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.129668    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-envoy" pod "b8574f16-35c1-4717-a83a-396210ca7a1f" (UID: "b8574f16-35c1-4717-a83a-396210ca7a1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:03.227937 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.227898    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vdjv\" (UniqueName: \"kubernetes.io/projected/b8574f16-35c1-4717-a83a-396210ca7a1f-kube-api-access-8vdjv\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:03.228124 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.227956    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:03.228124 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.227968    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:03.228124 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.227976    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8574f16-35c1-4717-a83a-396210ca7a1f-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:03.846793 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.846753    5053 generic.go:356] "Generic (PLEG): container finished" podID="b8574f16-35c1-4717-a83a-396210ca7a1f" containerID="66a8879dd6a4aa2f725f314ebc129836d6ebd079e40ea6319901894390e4ff69" exitCode=0
Nov 03 22:04:03.847291 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.846805    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"
Nov 03 22:04:03.847291 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.846828    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g" event={"ID":"b8574f16-35c1-4717-a83a-396210ca7a1f","Type":"ContainerDied","Data":"66a8879dd6a4aa2f725f314ebc129836d6ebd079e40ea6319901894390e4ff69"}
Nov 03 22:04:03.847291 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.846879    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g" event={"ID":"b8574f16-35c1-4717-a83a-396210ca7a1f","Type":"ContainerDied","Data":"ec38d20fe2f89aa2d39c2add1c554c4df5ddc25b422f7406e3b8bbc15458f8d9"}
Nov 03 22:04:03.847291 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.846901    5053 scope.go:122] "RemoveContainer" containerID="66a8879dd6a4aa2f725f314ebc129836d6ebd079e40ea6319901894390e4ff69"
Nov 03 22:04:03.857798 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.857776    5053 scope.go:122] "RemoveContainer" containerID="78cddba9c25d4cf6bb570f4c9e894c3c520919fe645ef8a1eca28f9e45fe8ed1"
Nov 03 22:04:03.869594 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.869576    5053 scope.go:122] "RemoveContainer" containerID="66a8879dd6a4aa2f725f314ebc129836d6ebd079e40ea6319901894390e4ff69"
Nov 03 22:04:03.869884 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:04:03.869856    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66a8879dd6a4aa2f725f314ebc129836d6ebd079e40ea6319901894390e4ff69\": container with ID starting with 66a8879dd6a4aa2f725f314ebc129836d6ebd079e40ea6319901894390e4ff69 not found: ID does not exist" containerID="66a8879dd6a4aa2f725f314ebc129836d6ebd079e40ea6319901894390e4ff69"
Nov 03 22:04:03.870011 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.869889    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66a8879dd6a4aa2f725f314ebc129836d6ebd079e40ea6319901894390e4ff69"} err="failed to get container status \"66a8879dd6a4aa2f725f314ebc129836d6ebd079e40ea6319901894390e4ff69\": rpc error: code = NotFound desc = could not find container \"66a8879dd6a4aa2f725f314ebc129836d6ebd079e40ea6319901894390e4ff69\": container with ID starting with 66a8879dd6a4aa2f725f314ebc129836d6ebd079e40ea6319901894390e4ff69 not found: ID does not exist"
Nov 03 22:04:03.870011 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.869908    5053 scope.go:122] "RemoveContainer" containerID="78cddba9c25d4cf6bb570f4c9e894c3c520919fe645ef8a1eca28f9e45fe8ed1"
Nov 03 22:04:03.870204 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:04:03.870184    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78cddba9c25d4cf6bb570f4c9e894c3c520919fe645ef8a1eca28f9e45fe8ed1\": container with ID starting with 78cddba9c25d4cf6bb570f4c9e894c3c520919fe645ef8a1eca28f9e45fe8ed1 not found: ID does not exist" containerID="78cddba9c25d4cf6bb570f4c9e894c3c520919fe645ef8a1eca28f9e45fe8ed1"
Nov 03 22:04:03.870310 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:03.870212    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78cddba9c25d4cf6bb570f4c9e894c3c520919fe645ef8a1eca28f9e45fe8ed1"} err="failed to get container status \"78cddba9c25d4cf6bb570f4c9e894c3c520919fe645ef8a1eca28f9e45fe8ed1\": rpc error: code = NotFound desc = could not find container \"78cddba9c25d4cf6bb570f4c9e894c3c520919fe645ef8a1eca28f9e45fe8ed1\": container with ID starting with 78cddba9c25d4cf6bb570f4c9e894c3c520919fe645ef8a1eca28f9e45fe8ed1 not found: ID does not exist"
Nov 03 22:04:04.014392 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:04.014355    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"]
Nov 03 22:04:04.064492 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:04.064457    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-7990bd-openshift-default-c484f4985-5bf6g"]
Nov 03 22:04:04.067592 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:04.067575    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74196013-crh2p"]
Nov 03 22:04:04.070440 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:04.070418    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74196013-crh2p"]
Nov 03 22:04:05.667101 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:05.667070    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c4fa70d-609b-481d-ae55-790ca51dd22b" path="/var/lib/kubelet/pods/3c4fa70d-609b-481d-ae55-790ca51dd22b/volumes"
Nov 03 22:04:05.667594 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:05.667577    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8574f16-35c1-4717-a83a-396210ca7a1f" path="/var/lib/kubelet/pods/b8574f16-35c1-4717-a83a-396210ca7a1f/volumes"
Nov 03 22:04:08.853128 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:08.853096    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"
Nov 03 22:04:10.936894 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:04:10.936845    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:04:38.788515 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:38.788481    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"]
Nov 03 22:04:38.789003 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:38.788707    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22" podUID="ed429be0-1f3f-4461-94d3-c4f18c389676" containerName="maas-api" containerID="cri-o://c6660f18f8b895525a0de0581418df4f731b3fd157f92848b00c66494ff06776" gracePeriod=30
Nov 03 22:04:39.110619 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.110591    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"
Nov 03 22:04:39.147383 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.147354    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed429be0-1f3f-4461-94d3-c4f18c389676-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed429be0-1f3f-4461-94d3-c4f18c389676-metrics-certs\") pod \"ed429be0-1f3f-4461-94d3-c4f18c389676\" (UID: \"ed429be0-1f3f-4461-94d3-c4f18c389676\") "
Nov 03 22:04:39.148344 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.148318    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed429be0-1f3f-4461-94d3-c4f18c389676-kube-api-access-96hxs\" (UniqueName: \"kubernetes.io/projected/ed429be0-1f3f-4461-94d3-c4f18c389676-kube-api-access-96hxs\") pod \"ed429be0-1f3f-4461-94d3-c4f18c389676\" (UID: \"ed429be0-1f3f-4461-94d3-c4f18c389676\") "
Nov 03 22:04:39.159455 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.159371    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed429be0-1f3f-4461-94d3-c4f18c389676-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed429be0-1f3f-4461-94d3-c4f18c389676-maas-api-tls\") pod \"ed429be0-1f3f-4461-94d3-c4f18c389676\" (UID: \"ed429be0-1f3f-4461-94d3-c4f18c389676\") "
Nov 03 22:04:39.159612 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.159459    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed429be0-1f3f-4461-94d3-c4f18c389676-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed429be0-1f3f-4461-94d3-c4f18c389676-openshift-service-ca\") pod \"ed429be0-1f3f-4461-94d3-c4f18c389676\" (UID: \"ed429be0-1f3f-4461-94d3-c4f18c389676\") "
Nov 03 22:04:39.160072 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.160027    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed429be0-1f3f-4461-94d3-c4f18c389676-openshift-service-ca" pod "ed429be0-1f3f-4461-94d3-c4f18c389676" (UID: "ed429be0-1f3f-4461-94d3-c4f18c389676"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:04:39.166728 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.163690    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed429be0-1f3f-4461-94d3-c4f18c389676-metrics-certs" pod "ed429be0-1f3f-4461-94d3-c4f18c389676" (UID: "ed429be0-1f3f-4461-94d3-c4f18c389676"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:04:39.171010 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.170976    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"]
Nov 03 22:04:39.171352 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.171226    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc" podUID="810ed10d-32b0-4e4f-ab41-37341380801a" containerName="istio-proxy" containerID="cri-o://138b80baec640feb7fb8c9c3de91fef691a1831ee538346d146e24a423956eef" gracePeriod=30
Nov 03 22:04:39.172678 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.172618    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed429be0-1f3f-4461-94d3-c4f18c389676-kube-api-access-96hxs" pod "ed429be0-1f3f-4461-94d3-c4f18c389676" (UID: "ed429be0-1f3f-4461-94d3-c4f18c389676"). InnerVolumeSpecName "kube-api-access-96hxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:04:39.172782 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.172707    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed429be0-1f3f-4461-94d3-c4f18c389676-maas-api-tls" pod "ed429be0-1f3f-4461-94d3-c4f18c389676" (UID: "ed429be0-1f3f-4461-94d3-c4f18c389676"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:04:39.189317 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.189289    5053 generic.go:356] "Generic (PLEG): container finished" podID="ed429be0-1f3f-4461-94d3-c4f18c389676" containerID="c6660f18f8b895525a0de0581418df4f731b3fd157f92848b00c66494ff06776" exitCode=0
Nov 03 22:04:39.189473 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.189343    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22" event={"ID":"ed429be0-1f3f-4461-94d3-c4f18c389676","Type":"ContainerDied","Data":"c6660f18f8b895525a0de0581418df4f731b3fd157f92848b00c66494ff06776"}
Nov 03 22:04:39.189473 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.189383    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22" event={"ID":"ed429be0-1f3f-4461-94d3-c4f18c389676","Type":"ContainerDied","Data":"84a87641be8a4effab771508f5f617e4e6c2a9f1cec4cf5c587181f62af94da1"}
Nov 03 22:04:39.189473 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.189404    5053 scope.go:122] "RemoveContainer" containerID="c6660f18f8b895525a0de0581418df4f731b3fd157f92848b00c66494ff06776"
Nov 03 22:04:39.189651 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.189543    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"
Nov 03 22:04:39.209052 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.209024    5053 scope.go:122] "RemoveContainer" containerID="c6660f18f8b895525a0de0581418df4f731b3fd157f92848b00c66494ff06776"
Nov 03 22:04:39.209431 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:04:39.209402    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6660f18f8b895525a0de0581418df4f731b3fd157f92848b00c66494ff06776\": container with ID starting with c6660f18f8b895525a0de0581418df4f731b3fd157f92848b00c66494ff06776 not found: ID does not exist" containerID="c6660f18f8b895525a0de0581418df4f731b3fd157f92848b00c66494ff06776"
Nov 03 22:04:39.209519 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.209446    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6660f18f8b895525a0de0581418df4f731b3fd157f92848b00c66494ff06776"} err="failed to get container status \"c6660f18f8b895525a0de0581418df4f731b3fd157f92848b00c66494ff06776\": rpc error: code = NotFound desc = could not find container \"c6660f18f8b895525a0de0581418df4f731b3fd157f92848b00c66494ff06776\": container with ID starting with c6660f18f8b895525a0de0581418df4f731b3fd157f92848b00c66494ff06776 not found: ID does not exist"
Nov 03 22:04:39.261328 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.261295    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed429be0-1f3f-4461-94d3-c4f18c389676-maas-api-tls\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:39.261328 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.261327    5053 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed429be0-1f3f-4461-94d3-c4f18c389676-openshift-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:39.261562 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.261339    5053 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed429be0-1f3f-4461-94d3-c4f18c389676-metrics-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:39.261562 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.261351    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96hxs\" (UniqueName: \"kubernetes.io/projected/ed429be0-1f3f-4461-94d3-c4f18c389676-kube-api-access-96hxs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:39.340486 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.340382    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"]
Nov 03 22:04:39.346937 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.346897    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22"]
Nov 03 22:04:39.667784 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:39.667709    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed429be0-1f3f-4461-94d3-c4f18c389676" path="/var/lib/kubelet/pods/ed429be0-1f3f-4461-94d3-c4f18c389676/volumes"
Nov 03 22:04:40.115699 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:40.115654    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66"]
Nov 03 22:04:40.116317 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:40.116302    5053 cpu_manager.go:414] "containerMap: removing container" podUID="b8574f16-35c1-4717-a83a-396210ca7a1f" containerName="istio-proxy"
Nov 03 22:04:40.116379 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:40.116318    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8574f16-35c1-4717-a83a-396210ca7a1f" containerName="istio-proxy"
Nov 03 22:04:40.116379 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:40.116350    5053 cpu_manager.go:414] "containerMap: removing container" podUID="ed429be0-1f3f-4461-94d3-c4f18c389676" containerName="maas-api"
Nov 03 22:04:40.116379 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:40.116356    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed429be0-1f3f-4461-94d3-c4f18c389676" containerName="maas-api"
Nov 03 22:04:40.116478 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:40.116472    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8574f16-35c1-4717-a83a-396210ca7a1f" containerName="istio-proxy"
Nov 03 22:04:40.116514 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:40.116486    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed429be0-1f3f-4461-94d3-c4f18c389676" containerName="maas-api"
Nov 03 22:04:40.121395 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:40.121377    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66"
Nov 03 22:04:40.175750 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:40.175712    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66"]
Nov 03 22:04:40.182047 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:40.182018    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njbxr\" (UniqueName: \"kubernetes.io/projected/6f4691fd-6104-4c05-a548-4fc331331b9b-kube-api-access-njbxr\") pod \"maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66\" (UID: \"6f4691fd-6104-4c05-a548-4fc331331b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66"
Nov 03 22:04:40.182213 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:40.182073    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4691fd-6104-4c05-a548-4fc331331b9b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66\" (UID: \"6f4691fd-6104-4c05-a548-4fc331331b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66"
Nov 03 22:04:40.182213 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:40.182192    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4691fd-6104-4c05-a548-4fc331331b9b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66\" (UID: \"6f4691fd-6104-4c05-a548-4fc331331b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66"
Nov 03 22:04:40.220792 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:40.220753    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:04:40.220792 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:40.220758    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:04:40.283573 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:40.283544    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njbxr\" (UniqueName: \"kubernetes.io/projected/6f4691fd-6104-4c05-a548-4fc331331b9b-kube-api-access-njbxr\") pod \"maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66\" (UID: \"6f4691fd-6104-4c05-a548-4fc331331b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66"
Nov 03 22:04:40.283760 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:40.283596    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4691fd-6104-4c05-a548-4fc331331b9b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66\" (UID: \"6f4691fd-6104-4c05-a548-4fc331331b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66"
Nov 03 22:04:40.283760 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:40.283657    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4691fd-6104-4c05-a548-4fc331331b9b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66\" (UID: \"6f4691fd-6104-4c05-a548-4fc331331b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66"
Nov 03 22:04:40.284283 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:40.284258    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4691fd-6104-4c05-a548-4fc331331b9b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66\" (UID: \"6f4691fd-6104-4c05-a548-4fc331331b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66"
Nov 03 22:04:40.286141 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:40.286121    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4691fd-6104-4c05-a548-4fc331331b9b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66\" (UID: \"6f4691fd-6104-4c05-a548-4fc331331b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66"
Nov 03 22:04:40.339952 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:40.339920    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njbxr\" (UniqueName: \"kubernetes.io/projected/6f4691fd-6104-4c05-a548-4fc331331b9b-kube-api-access-njbxr\") pod \"maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66\" (UID: \"6f4691fd-6104-4c05-a548-4fc331331b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66"
Nov 03 22:04:40.435249 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:40.435159    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66"
Nov 03 22:04:40.819301 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:40.816604    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66"]
Nov 03 22:04:40.841738 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:04:40.841702    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4691fd_6104_4c05_a548_4fc331331b9b.slice/crio-6888002d193438b9ed7f7624cd13acdc1a97c57392d0ec422a79c0117e8edc5b WatchSource:0}: Error finding container 6888002d193438b9ed7f7624cd13acdc1a97c57392d0ec422a79c0117e8edc5b: Status 404 returned error can't find the container with id 6888002d193438b9ed7f7624cd13acdc1a97c57392d0ec422a79c0117e8edc5b
Nov 03 22:04:41.046312 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:04:41.046273    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:04:41.185331 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:41.185197    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n"]
Nov 03 22:04:41.207585 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:41.207549    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66" event={"ID":"6f4691fd-6104-4c05-a548-4fc331331b9b","Type":"ContainerStarted","Data":"6888002d193438b9ed7f7624cd13acdc1a97c57392d0ec422a79c0117e8edc5b"}
Nov 03 22:04:41.234496 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:41.234459    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bd35c-5hd8n"]
Nov 03 22:04:41.667921 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:41.667892    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4030fc07-5675-4b89-bb00-4f026cebb1b4" path="/var/lib/kubelet/pods/4030fc07-5675-4b89-bb00-4f026cebb1b4/volumes"
Nov 03 22:04:42.037905 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:42.037853    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bd35c-b9785c584-wmd22" podUID="ed429be0-1f3f-4461-94d3-c4f18c389676" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.215:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:04:42.217159 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:42.217119    5053 generic.go:356] "Generic (PLEG): container finished" podID="6f4691fd-6104-4c05-a548-4fc331331b9b" containerID="d084b1000efa923450ba423af869c04b5365a8b9408265b8ea715081ef0f7e85" exitCode=0
Nov 03 22:04:42.217700 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:42.217202    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66" event={"ID":"6f4691fd-6104-4c05-a548-4fc331331b9b","Type":"ContainerDied","Data":"d084b1000efa923450ba423af869c04b5365a8b9408265b8ea715081ef0f7e85"}
Nov 03 22:04:43.390898 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:43.390868    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66"
Nov 03 22:04:43.531574 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:43.531462    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4691fd-6104-4c05-a548-4fc331331b9b-kube-api-access-njbxr\" (UniqueName: \"kubernetes.io/projected/6f4691fd-6104-4c05-a548-4fc331331b9b-kube-api-access-njbxr\") pod \"6f4691fd-6104-4c05-a548-4fc331331b9b\" (UID: \"6f4691fd-6104-4c05-a548-4fc331331b9b\") "
Nov 03 22:04:43.531574 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:43.531542    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f4691fd-6104-4c05-a548-4fc331331b9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4691fd-6104-4c05-a548-4fc331331b9b-maas-api-openshift-ca\") pod \"6f4691fd-6104-4c05-a548-4fc331331b9b\" (UID: \"6f4691fd-6104-4c05-a548-4fc331331b9b\") "
Nov 03 22:04:43.531831 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:43.531611    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f4691fd-6104-4c05-a548-4fc331331b9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4691fd-6104-4c05-a548-4fc331331b9b-maas-api-xks-ca\") pod \"6f4691fd-6104-4c05-a548-4fc331331b9b\" (UID: \"6f4691fd-6104-4c05-a548-4fc331331b9b\") "
Nov 03 22:04:43.531989 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:43.531958    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4691fd-6104-4c05-a548-4fc331331b9b-maas-api-openshift-ca" pod "6f4691fd-6104-4c05-a548-4fc331331b9b" (UID: "6f4691fd-6104-4c05-a548-4fc331331b9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:04:43.532403 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:43.532380    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4691fd-6104-4c05-a548-4fc331331b9b-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:43.534825 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:43.534794    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f4691fd-6104-4c05-a548-4fc331331b9b-maas-api-xks-ca" pod "6f4691fd-6104-4c05-a548-4fc331331b9b" (UID: "6f4691fd-6104-4c05-a548-4fc331331b9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:04:43.534947 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:43.534851    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4691fd-6104-4c05-a548-4fc331331b9b-kube-api-access-njbxr" pod "6f4691fd-6104-4c05-a548-4fc331331b9b" (UID: "6f4691fd-6104-4c05-a548-4fc331331b9b"). InnerVolumeSpecName "kube-api-access-njbxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:04:43.633907 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:43.633859    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njbxr\" (UniqueName: \"kubernetes.io/projected/6f4691fd-6104-4c05-a548-4fc331331b9b-kube-api-access-njbxr\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:43.633907 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:43.633898    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4691fd-6104-4c05-a548-4fc331331b9b-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:44.234495 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.234469    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66"
Nov 03 22:04:44.234495 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.234474    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66" event={"ID":"6f4691fd-6104-4c05-a548-4fc331331b9b","Type":"ContainerDied","Data":"6888002d193438b9ed7f7624cd13acdc1a97c57392d0ec422a79c0117e8edc5b"}
Nov 03 22:04:44.234685 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.234526    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6888002d193438b9ed7f7624cd13acdc1a97c57392d0ec422a79c0117e8edc5b"
Nov 03 22:04:44.445747 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.445716    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 22:04:44.546350 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.546315    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-istio-data\") pod \"810ed10d-32b0-4e4f-ab41-37341380801a\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") "
Nov 03 22:04:44.546539 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.546400    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-istio-envoy\") pod \"810ed10d-32b0-4e4f-ab41-37341380801a\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") "
Nov 03 22:04:44.546539 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.546475    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-credential-socket\") pod \"810ed10d-32b0-4e4f-ab41-37341380801a\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") "
Nov 03 22:04:44.546646 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.546555    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-workload-certs\") pod \"810ed10d-32b0-4e4f-ab41-37341380801a\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") "
Nov 03 22:04:44.546646 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.546606    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-istio-data" pod "810ed10d-32b0-4e4f-ab41-37341380801a" (UID: "810ed10d-32b0-4e4f-ab41-37341380801a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:44.546646 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.546619    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/810ed10d-32b0-4e4f-ab41-37341380801a-kube-api-access-bwgvg\" (UniqueName: \"kubernetes.io/projected/810ed10d-32b0-4e4f-ab41-37341380801a-kube-api-access-bwgvg\") pod \"810ed10d-32b0-4e4f-ab41-37341380801a\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") "
Nov 03 22:04:44.546790 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.546659    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/810ed10d-32b0-4e4f-ab41-37341380801a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/810ed10d-32b0-4e4f-ab41-37341380801a-istiod-ca-cert\") pod \"810ed10d-32b0-4e4f-ab41-37341380801a\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") "
Nov 03 22:04:44.546790 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.546689    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/810ed10d-32b0-4e4f-ab41-37341380801a-istio-token\" (UniqueName: \"kubernetes.io/projected/810ed10d-32b0-4e4f-ab41-37341380801a-istio-token\") pod \"810ed10d-32b0-4e4f-ab41-37341380801a\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") "
Nov 03 22:04:44.546790 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.546731    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/810ed10d-32b0-4e4f-ab41-37341380801a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/810ed10d-32b0-4e4f-ab41-37341380801a-istio-podinfo\") pod \"810ed10d-32b0-4e4f-ab41-37341380801a\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") "
Nov 03 22:04:44.546958 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.546835    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-workload-socket\") pod \"810ed10d-32b0-4e4f-ab41-37341380801a\" (UID: \"810ed10d-32b0-4e4f-ab41-37341380801a\") "
Nov 03 22:04:44.546958 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.546851    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-credential-socket" pod "810ed10d-32b0-4e4f-ab41-37341380801a" (UID: "810ed10d-32b0-4e4f-ab41-37341380801a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:44.546958 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.546865    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-workload-certs" pod "810ed10d-32b0-4e4f-ab41-37341380801a" (UID: "810ed10d-32b0-4e4f-ab41-37341380801a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:44.547147 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.547112    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-workload-socket" pod "810ed10d-32b0-4e4f-ab41-37341380801a" (UID: "810ed10d-32b0-4e4f-ab41-37341380801a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:44.547211 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.547140    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/810ed10d-32b0-4e4f-ab41-37341380801a-istiod-ca-cert" pod "810ed10d-32b0-4e4f-ab41-37341380801a" (UID: "810ed10d-32b0-4e4f-ab41-37341380801a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:04:44.547625 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.547540    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:44.547625 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.547566    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:44.547625 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.547580    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:44.547625 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.547593    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:44.547625 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.547606    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/810ed10d-32b0-4e4f-ab41-37341380801a-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:44.549227 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.549194    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/810ed10d-32b0-4e4f-ab41-37341380801a-istio-podinfo" pod "810ed10d-32b0-4e4f-ab41-37341380801a" (UID: "810ed10d-32b0-4e4f-ab41-37341380801a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:04:44.549368 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.549220    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/810ed10d-32b0-4e4f-ab41-37341380801a-kube-api-access-bwgvg" pod "810ed10d-32b0-4e4f-ab41-37341380801a" (UID: "810ed10d-32b0-4e4f-ab41-37341380801a"). InnerVolumeSpecName "kube-api-access-bwgvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:04:44.549470 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.549450    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/810ed10d-32b0-4e4f-ab41-37341380801a-istio-token" pod "810ed10d-32b0-4e4f-ab41-37341380801a" (UID: "810ed10d-32b0-4e4f-ab41-37341380801a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:04:44.549551 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.549534    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-istio-envoy" pod "810ed10d-32b0-4e4f-ab41-37341380801a" (UID: "810ed10d-32b0-4e4f-ab41-37341380801a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:44.648949 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.648916    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/810ed10d-32b0-4e4f-ab41-37341380801a-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:44.648949 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.648944    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwgvg\" (UniqueName: \"kubernetes.io/projected/810ed10d-32b0-4e4f-ab41-37341380801a-kube-api-access-bwgvg\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:44.648949 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.648953    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/810ed10d-32b0-4e4f-ab41-37341380801a-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:44.649192 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:44.648962    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/810ed10d-32b0-4e4f-ab41-37341380801a-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:45.276262 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:45.262835    5053 generic.go:356] "Generic (PLEG): container finished" podID="810ed10d-32b0-4e4f-ab41-37341380801a" containerID="138b80baec640feb7fb8c9c3de91fef691a1831ee538346d146e24a423956eef" exitCode=0
Nov 03 22:04:45.276262 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:45.263162    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc" event={"ID":"810ed10d-32b0-4e4f-ab41-37341380801a","Type":"ContainerDied","Data":"138b80baec640feb7fb8c9c3de91fef691a1831ee538346d146e24a423956eef"}
Nov 03 22:04:45.276262 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:45.263264    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc" event={"ID":"810ed10d-32b0-4e4f-ab41-37341380801a","Type":"ContainerDied","Data":"3c3f3347d56385ab8f682fe6f9c96e42fbca5d11bc97a63da31f4cac55bd9fab"}
Nov 03 22:04:45.276262 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:45.263304    5053 scope.go:122] "RemoveContainer" containerID="138b80baec640feb7fb8c9c3de91fef691a1831ee538346d146e24a423956eef"
Nov 03 22:04:45.276262 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:45.263556    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"
Nov 03 22:04:45.315679 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:45.310777    5053 scope.go:122] "RemoveContainer" containerID="f1b8a8ce5d90f4c4473002c9ab8f121b30eebc99fe1d5d37d8c5892b913c2094"
Nov 03 22:04:45.327599 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:45.327572    5053 scope.go:122] "RemoveContainer" containerID="138b80baec640feb7fb8c9c3de91fef691a1831ee538346d146e24a423956eef"
Nov 03 22:04:45.327934 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:04:45.327912    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"138b80baec640feb7fb8c9c3de91fef691a1831ee538346d146e24a423956eef\": container with ID starting with 138b80baec640feb7fb8c9c3de91fef691a1831ee538346d146e24a423956eef not found: ID does not exist" containerID="138b80baec640feb7fb8c9c3de91fef691a1831ee538346d146e24a423956eef"
Nov 03 22:04:45.328016 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:45.327943    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"138b80baec640feb7fb8c9c3de91fef691a1831ee538346d146e24a423956eef"} err="failed to get container status \"138b80baec640feb7fb8c9c3de91fef691a1831ee538346d146e24a423956eef\": rpc error: code = NotFound desc = could not find container \"138b80baec640feb7fb8c9c3de91fef691a1831ee538346d146e24a423956eef\": container with ID starting with 138b80baec640feb7fb8c9c3de91fef691a1831ee538346d146e24a423956eef not found: ID does not exist"
Nov 03 22:04:45.328016 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:45.327960    5053 scope.go:122] "RemoveContainer" containerID="f1b8a8ce5d90f4c4473002c9ab8f121b30eebc99fe1d5d37d8c5892b913c2094"
Nov 03 22:04:45.328326 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:04:45.328265    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1b8a8ce5d90f4c4473002c9ab8f121b30eebc99fe1d5d37d8c5892b913c2094\": container with ID starting with f1b8a8ce5d90f4c4473002c9ab8f121b30eebc99fe1d5d37d8c5892b913c2094 not found: ID does not exist" containerID="f1b8a8ce5d90f4c4473002c9ab8f121b30eebc99fe1d5d37d8c5892b913c2094"
Nov 03 22:04:45.328326 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:45.328297    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b8a8ce5d90f4c4473002c9ab8f121b30eebc99fe1d5d37d8c5892b913c2094"} err="failed to get container status \"f1b8a8ce5d90f4c4473002c9ab8f121b30eebc99fe1d5d37d8c5892b913c2094\": rpc error: code = NotFound desc = could not find container \"f1b8a8ce5d90f4c4473002c9ab8f121b30eebc99fe1d5d37d8c5892b913c2094\": container with ID starting with f1b8a8ce5d90f4c4473002c9ab8f121b30eebc99fe1d5d37d8c5892b913c2094 not found: ID does not exist"
Nov 03 22:04:45.336543 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:04:45.336516    5053 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810ed10d_32b0_4e4f_ab41_37341380801a.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 22:04:45.429995 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:45.429959    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"]
Nov 03 22:04:45.479808 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:45.479773    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-eb3520-openshift-default-69c56bd79c-c42lc"]
Nov 03 22:04:45.667804 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:45.667724    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="810ed10d-32b0-4e4f-ab41-37341380801a" path="/var/lib/kubelet/pods/810ed10d-32b0-4e4f-ab41-37341380801a/volumes"
Nov 03 22:04:46.183285 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:46.183256    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 22:04:46.187694 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:46.187674    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 22:04:46.193507 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:46.193490    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 22:04:46.197969 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:46.197948    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 22:04:47.779282 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:47.779214    5053 scope.go:122] "RemoveContainer" containerID="ce97895737408d7f79eec730b139bfaa480270b1edabd40bf5112810ba4e490a"
Nov 03 22:04:49.482617 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.482578    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"]
Nov 03 22:04:49.483332 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.483315    5053 cpu_manager.go:414] "containerMap: removing container" podUID="b8574f16-35c1-4717-a83a-396210ca7a1f" containerName="istio-proxy"
Nov 03 22:04:49.483398 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.483335    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8574f16-35c1-4717-a83a-396210ca7a1f" containerName="istio-proxy"
Nov 03 22:04:49.483398 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.483345    5053 cpu_manager.go:414] "containerMap: removing container" podUID="6f4691fd-6104-4c05-a548-4fc331331b9b" containerName="revoke-keys"
Nov 03 22:04:49.483398 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.483350    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f4691fd-6104-4c05-a548-4fc331331b9b" containerName="revoke-keys"
Nov 03 22:04:49.483398 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.483364    5053 cpu_manager.go:414] "containerMap: removing container" podUID="810ed10d-32b0-4e4f-ab41-37341380801a" containerName="istio-proxy"
Nov 03 22:04:49.483398 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.483369    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="810ed10d-32b0-4e4f-ab41-37341380801a" containerName="istio-proxy"
Nov 03 22:04:49.483398 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.483381    5053 cpu_manager.go:414] "containerMap: removing container" podUID="810ed10d-32b0-4e4f-ab41-37341380801a" containerName="istio-proxy"
Nov 03 22:04:49.483398 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.483385    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="810ed10d-32b0-4e4f-ab41-37341380801a" containerName="istio-proxy"
Nov 03 22:04:49.483628 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.483505    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8574f16-35c1-4717-a83a-396210ca7a1f" containerName="istio-proxy"
Nov 03 22:04:49.483628 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.483513    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f4691fd-6104-4c05-a548-4fc331331b9b" containerName="revoke-keys"
Nov 03 22:04:49.483628 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.483519    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="810ed10d-32b0-4e4f-ab41-37341380801a" containerName="istio-proxy"
Nov 03 22:04:49.483628 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.483526    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="810ed10d-32b0-4e4f-ab41-37341380801a" containerName="istio-proxy"
Nov 03 22:04:49.489453 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.489434    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.566263 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.564435    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-0d45ba-openshift-default-dockercfg-w5vdf\""
Nov 03 22:04:49.645688 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.645635    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-workload-socket\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.646053 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.646033    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-token\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.646277 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.646252    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lprhz\" (UniqueName: \"kubernetes.io/projected/0c981726-d50f-4694-93d4-ebefb45bfb47-kube-api-access-lprhz\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.646501 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.646478    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-workload-certs\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.646729 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.646713    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-data\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.646900 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.646877    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-podinfo\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.647025 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.647008    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-credential-socket\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.647152 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.647136    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c981726-d50f-4694-93d4-ebefb45bfb47-istiod-ca-cert\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.647336 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.647314    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-envoy\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.713279 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.704344    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"]
Nov 03 22:04:49.749266 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.749169    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-podinfo\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.749504 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.749481    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-credential-socket\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.749632 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.749617    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c981726-d50f-4694-93d4-ebefb45bfb47-istiod-ca-cert\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.749783 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.749768    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-envoy\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.749941 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.749926    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-workload-socket\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.750060 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.750049    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-token\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.750184 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.750167    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lprhz\" (UniqueName: \"kubernetes.io/projected/0c981726-d50f-4694-93d4-ebefb45bfb47-kube-api-access-lprhz\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.750408 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.750390    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-workload-certs\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.750611 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.750591    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-data\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.750997 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.750982    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-data\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.751309 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.751295    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-workload-socket\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.751997 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.751982    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-workload-certs\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.752304 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.752292    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-credential-socket\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.752848 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.752833    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c981726-d50f-4694-93d4-ebefb45bfb47-istiod-ca-cert\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.754838 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.754811    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-podinfo\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.768267 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.764502    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-envoy\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.808435 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.808404    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-token\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:49.808625 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:49.808513    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lprhz\" (UniqueName: \"kubernetes.io/projected/0c981726-d50f-4694-93d4-ebefb45bfb47-kube-api-access-lprhz\") pod \"e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") " pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:50.103742 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:50.103649    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:50.542718 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:50.542681    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"]
Nov 03 22:04:50.548304 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:04:50.548221    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c981726_d50f_4694_93d4_ebefb45bfb47.slice/crio-997db12c76c76a0d1e6c74d4ea73bf937181f7e09b3ce37f1acbacb5c10f6919 WatchSource:0}: Error finding container 997db12c76c76a0d1e6c74d4ea73bf937181f7e09b3ce37f1acbacb5c10f6919: Status 404 returned error can't find the container with id 997db12c76c76a0d1e6c74d4ea73bf937181f7e09b3ce37f1acbacb5c10f6919
Nov 03 22:04:50.551906 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:50.551857    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:04:50.552040 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:50.551946    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:04:50.552040 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:50.551983    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:04:51.327176 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:51.327135    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf" event={"ID":"0c981726-d50f-4694-93d4-ebefb45bfb47","Type":"ContainerStarted","Data":"0c20c4d038d8d2aba7c4324a914c750bb2fd8aa17bcea5c98b7140aafe24eb51"}
Nov 03 22:04:51.327446 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:51.327200    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf" event={"ID":"0c981726-d50f-4694-93d4-ebefb45bfb47","Type":"ContainerStarted","Data":"997db12c76c76a0d1e6c74d4ea73bf937181f7e09b3ce37f1acbacb5c10f6919"}
Nov 03 22:04:51.446503 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:51.446435    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf" podStartSLOduration=2.446416576 podStartE2EDuration="2.446416576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:04:51.441945666 +0000 UTC m=+3005.935867540" watchObservedRunningTime="2026-11-03 22:04:51.446416576 +0000 UTC m=+3005.940338452"
Nov 03 22:04:52.104140 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:52.104098    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:52.110139 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:52.110112    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:52.334501 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:52.334468    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:52.335796 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:52.335775    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:04:55.638245 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:55.638159    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66"]
Nov 03 22:04:55.688253 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:55.688203    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66"]
Nov 03 22:04:55.715855 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:04:55.715814    5053 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66\" is forbidden: User \"system:node:ip-10-0-128-43.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-43.us-west-1.compute.internal' and this object" podUID="6f4691fd-6104-4c05-a548-4fc331331b9b" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-95d604-m7v66"
Nov 03 22:04:57.669158 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:04:57.669126    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f4691fd-6104-4c05-a548-4fc331331b9b" path="/var/lib/kubelet/pods/6f4691fd-6104-4c05-a548-4fc331331b9b/volumes"
Nov 03 22:05:02.894763 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:02.894727    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"]
Nov 03 22:05:02.895184 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:02.894960    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd" podUID="122b09a6-7732-417d-b82f-a977510c180f" containerName="istio-proxy" containerID="cri-o://6b75ded44a8b59d2f9c9a809f315c0372e4dc771a8b3f6c8025de9314e90e473" gracePeriod=30
Nov 03 22:05:08.177194 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.177132    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 22:05:08.202644 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.202614    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-workload-certs\") pod \"122b09a6-7732-417d-b82f-a977510c180f\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") "
Nov 03 22:05:08.202809 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.202658    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/122b09a6-7732-417d-b82f-a977510c180f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/122b09a6-7732-417d-b82f-a977510c180f-istiod-ca-cert\") pod \"122b09a6-7732-417d-b82f-a977510c180f\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") "
Nov 03 22:05:08.202809 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.202694    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/122b09a6-7732-417d-b82f-a977510c180f-kube-api-access-5nw54\" (UniqueName: \"kubernetes.io/projected/122b09a6-7732-417d-b82f-a977510c180f-kube-api-access-5nw54\") pod \"122b09a6-7732-417d-b82f-a977510c180f\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") "
Nov 03 22:05:08.202809 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.202763    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/122b09a6-7732-417d-b82f-a977510c180f-istio-token\" (UniqueName: \"kubernetes.io/projected/122b09a6-7732-417d-b82f-a977510c180f-istio-token\") pod \"122b09a6-7732-417d-b82f-a977510c180f\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") "
Nov 03 22:05:08.203001 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.202816    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-istio-envoy\") pod \"122b09a6-7732-417d-b82f-a977510c180f\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") "
Nov 03 22:05:08.203001 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.202856    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-workload-socket\") pod \"122b09a6-7732-417d-b82f-a977510c180f\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") "
Nov 03 22:05:08.203001 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.202906    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-workload-certs" pod "122b09a6-7732-417d-b82f-a977510c180f" (UID: "122b09a6-7732-417d-b82f-a977510c180f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:05:08.203001 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.202922    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-credential-socket\") pod \"122b09a6-7732-417d-b82f-a977510c180f\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") "
Nov 03 22:05:08.203229 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.203076    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-istio-data\") pod \"122b09a6-7732-417d-b82f-a977510c180f\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") "
Nov 03 22:05:08.203229 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.203154    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/122b09a6-7732-417d-b82f-a977510c180f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/122b09a6-7732-417d-b82f-a977510c180f-istio-podinfo\") pod \"122b09a6-7732-417d-b82f-a977510c180f\" (UID: \"122b09a6-7732-417d-b82f-a977510c180f\") "
Nov 03 22:05:08.203229 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.203161    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-credential-socket" pod "122b09a6-7732-417d-b82f-a977510c180f" (UID: "122b09a6-7732-417d-b82f-a977510c180f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:05:08.203229 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.203173    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/122b09a6-7732-417d-b82f-a977510c180f-istiod-ca-cert" pod "122b09a6-7732-417d-b82f-a977510c180f" (UID: "122b09a6-7732-417d-b82f-a977510c180f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:05:08.203763 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.203711    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-istio-data" pod "122b09a6-7732-417d-b82f-a977510c180f" (UID: "122b09a6-7732-417d-b82f-a977510c180f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:05:08.203864 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.203855    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:05:08.203932 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.203873    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/122b09a6-7732-417d-b82f-a977510c180f-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:05:08.203932 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.203887    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:05:08.203932 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.203900    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:05:08.204206 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.204165    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-workload-socket" pod "122b09a6-7732-417d-b82f-a977510c180f" (UID: "122b09a6-7732-417d-b82f-a977510c180f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:05:08.205980 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.205943    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122b09a6-7732-417d-b82f-a977510c180f-kube-api-access-5nw54" pod "122b09a6-7732-417d-b82f-a977510c180f" (UID: "122b09a6-7732-417d-b82f-a977510c180f"). InnerVolumeSpecName "kube-api-access-5nw54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:05:08.206223 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.206188    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122b09a6-7732-417d-b82f-a977510c180f-istio-token" pod "122b09a6-7732-417d-b82f-a977510c180f" (UID: "122b09a6-7732-417d-b82f-a977510c180f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:05:08.206381 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.206222    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/122b09a6-7732-417d-b82f-a977510c180f-istio-podinfo" pod "122b09a6-7732-417d-b82f-a977510c180f" (UID: "122b09a6-7732-417d-b82f-a977510c180f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:05:08.206444 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.206388    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-istio-envoy" pod "122b09a6-7732-417d-b82f-a977510c180f" (UID: "122b09a6-7732-417d-b82f-a977510c180f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:05:08.305860 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.305817    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nw54\" (UniqueName: \"kubernetes.io/projected/122b09a6-7732-417d-b82f-a977510c180f-kube-api-access-5nw54\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:05:08.305860 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.305857    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/122b09a6-7732-417d-b82f-a977510c180f-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:05:08.306123 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.305877    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:05:08.306123 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.305891    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/122b09a6-7732-417d-b82f-a977510c180f-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:05:08.306123 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.305903    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/122b09a6-7732-417d-b82f-a977510c180f-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:05:08.451894 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.451853    5053 generic.go:356] "Generic (PLEG): container finished" podID="122b09a6-7732-417d-b82f-a977510c180f" containerID="6b75ded44a8b59d2f9c9a809f315c0372e4dc771a8b3f6c8025de9314e90e473" exitCode=0
Nov 03 22:05:08.451894 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.451896    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd" event={"ID":"122b09a6-7732-417d-b82f-a977510c180f","Type":"ContainerDied","Data":"6b75ded44a8b59d2f9c9a809f315c0372e4dc771a8b3f6c8025de9314e90e473"}
Nov 03 22:05:08.452175 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.451942    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"
Nov 03 22:05:08.452175 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.451973    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd" event={"ID":"122b09a6-7732-417d-b82f-a977510c180f","Type":"ContainerDied","Data":"c0914d56429506ab2a1255f5da2aeba8828a23a6e170000dcd25a4312fc6bfba"}
Nov 03 22:05:08.452175 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.451996    5053 scope.go:122] "RemoveContainer" containerID="6b75ded44a8b59d2f9c9a809f315c0372e4dc771a8b3f6c8025de9314e90e473"
Nov 03 22:05:08.463789 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.463767    5053 scope.go:122] "RemoveContainer" containerID="292948e4406f710ebace8840ca6143130122c4774da6958deb001b408cc68ebe"
Nov 03 22:05:08.473389 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.473369    5053 scope.go:122] "RemoveContainer" containerID="6b75ded44a8b59d2f9c9a809f315c0372e4dc771a8b3f6c8025de9314e90e473"
Nov 03 22:05:08.473681 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:05:08.473660    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b75ded44a8b59d2f9c9a809f315c0372e4dc771a8b3f6c8025de9314e90e473\": container with ID starting with 6b75ded44a8b59d2f9c9a809f315c0372e4dc771a8b3f6c8025de9314e90e473 not found: ID does not exist" containerID="6b75ded44a8b59d2f9c9a809f315c0372e4dc771a8b3f6c8025de9314e90e473"
Nov 03 22:05:08.473724 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.473692    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b75ded44a8b59d2f9c9a809f315c0372e4dc771a8b3f6c8025de9314e90e473"} err="failed to get container status \"6b75ded44a8b59d2f9c9a809f315c0372e4dc771a8b3f6c8025de9314e90e473\": rpc error: code = NotFound desc = could not find container \"6b75ded44a8b59d2f9c9a809f315c0372e4dc771a8b3f6c8025de9314e90e473\": container with ID starting with 6b75ded44a8b59d2f9c9a809f315c0372e4dc771a8b3f6c8025de9314e90e473 not found: ID does not exist"
Nov 03 22:05:08.473724 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.473710    5053 scope.go:122] "RemoveContainer" containerID="292948e4406f710ebace8840ca6143130122c4774da6958deb001b408cc68ebe"
Nov 03 22:05:08.473950 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:05:08.473929    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"292948e4406f710ebace8840ca6143130122c4774da6958deb001b408cc68ebe\": container with ID starting with 292948e4406f710ebace8840ca6143130122c4774da6958deb001b408cc68ebe not found: ID does not exist" containerID="292948e4406f710ebace8840ca6143130122c4774da6958deb001b408cc68ebe"
Nov 03 22:05:08.473991 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.473956    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292948e4406f710ebace8840ca6143130122c4774da6958deb001b408cc68ebe"} err="failed to get container status \"292948e4406f710ebace8840ca6143130122c4774da6958deb001b408cc68ebe\": rpc error: code = NotFound desc = could not find container \"292948e4406f710ebace8840ca6143130122c4774da6958deb001b408cc68ebe\": container with ID starting with 292948e4406f710ebace8840ca6143130122c4774da6958deb001b408cc68ebe not found: ID does not exist"
Nov 03 22:05:08.618884 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.618846    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"]
Nov 03 22:05:08.623784 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.623757    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-95d604-openshift-default-8658687cfc-vhhcd"]
Nov 03 22:05:08.702412 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.702326    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b97c67-pl9qn"]
Nov 03 22:05:08.703013 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.702995    5053 cpu_manager.go:414] "containerMap: removing container" podUID="122b09a6-7732-417d-b82f-a977510c180f" containerName="istio-proxy"
Nov 03 22:05:08.703013 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.703014    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="122b09a6-7732-417d-b82f-a977510c180f" containerName="istio-proxy"
Nov 03 22:05:08.703163 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.703122    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="122b09a6-7732-417d-b82f-a977510c180f" containerName="istio-proxy"
Nov 03 22:05:08.708348 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.708327    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b97c67-pl9qn"
Nov 03 22:05:08.760160 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.760129    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:05:08.760833 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.760808    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:05:08.761622 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.761598    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b97c67-pl9qn"]
Nov 03 22:05:08.813533 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.813504    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07179fa-cced-46f1-8952-8fe700769c7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6b97c67-pl9qn\" (UID: \"b07179fa-cced-46f1-8952-8fe700769c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b97c67-pl9qn"
Nov 03 22:05:08.813725 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.813540    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvr8g\" (UniqueName: \"kubernetes.io/projected/b07179fa-cced-46f1-8952-8fe700769c7a-kube-api-access-lvr8g\") pod \"maas-api-revoke-sub-d6b97c67-pl9qn\" (UID: \"b07179fa-cced-46f1-8952-8fe700769c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b97c67-pl9qn"
Nov 03 22:05:08.813725 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.813668    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07179fa-cced-46f1-8952-8fe700769c7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6b97c67-pl9qn\" (UID: \"b07179fa-cced-46f1-8952-8fe700769c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b97c67-pl9qn"
Nov 03 22:05:08.915444 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.915413    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07179fa-cced-46f1-8952-8fe700769c7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6b97c67-pl9qn\" (UID: \"b07179fa-cced-46f1-8952-8fe700769c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b97c67-pl9qn"
Nov 03 22:05:08.915666 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.915455    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvr8g\" (UniqueName: \"kubernetes.io/projected/b07179fa-cced-46f1-8952-8fe700769c7a-kube-api-access-lvr8g\") pod \"maas-api-revoke-sub-d6b97c67-pl9qn\" (UID: \"b07179fa-cced-46f1-8952-8fe700769c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b97c67-pl9qn"
Nov 03 22:05:08.915666 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.915508    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07179fa-cced-46f1-8952-8fe700769c7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6b97c67-pl9qn\" (UID: \"b07179fa-cced-46f1-8952-8fe700769c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b97c67-pl9qn"
Nov 03 22:05:08.916121 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.916093    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07179fa-cced-46f1-8952-8fe700769c7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6b97c67-pl9qn\" (UID: \"b07179fa-cced-46f1-8952-8fe700769c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b97c67-pl9qn"
Nov 03 22:05:08.918017 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.918000    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07179fa-cced-46f1-8952-8fe700769c7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6b97c67-pl9qn\" (UID: \"b07179fa-cced-46f1-8952-8fe700769c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b97c67-pl9qn"
Nov 03 22:05:08.972618 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:08.972547    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvr8g\" (UniqueName: \"kubernetes.io/projected/b07179fa-cced-46f1-8952-8fe700769c7a-kube-api-access-lvr8g\") pod \"maas-api-revoke-sub-d6b97c67-pl9qn\" (UID: \"b07179fa-cced-46f1-8952-8fe700769c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b97c67-pl9qn"
Nov 03 22:05:09.024888 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:09.024856    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b97c67-pl9qn"
Nov 03 22:05:09.402505 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:09.402478    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b97c67-pl9qn"]
Nov 03 22:05:09.403803 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:05:09.403773    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07179fa_cced_46f1_8952_8fe700769c7a.slice/crio-b09fc0c6e87a89a572dbf450863da92f69bda9362ad7b3b2f526f754d30729ca WatchSource:0}: Error finding container b09fc0c6e87a89a572dbf450863da92f69bda9362ad7b3b2f526f754d30729ca: Status 404 returned error can't find the container with id b09fc0c6e87a89a572dbf450863da92f69bda9362ad7b3b2f526f754d30729ca
Nov 03 22:05:09.460684 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:09.460656    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b97c67-pl9qn" event={"ID":"b07179fa-cced-46f1-8952-8fe700769c7a","Type":"ContainerStarted","Data":"b09fc0c6e87a89a572dbf450863da92f69bda9362ad7b3b2f526f754d30729ca"}
Nov 03 22:05:09.670932 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:09.670855    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="122b09a6-7732-417d-b82f-a977510c180f" path="/var/lib/kubelet/pods/122b09a6-7732-417d-b82f-a977510c180f/volumes"
Nov 03 22:05:10.469015 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:10.468976    5053 generic.go:356] "Generic (PLEG): container finished" podID="b07179fa-cced-46f1-8952-8fe700769c7a" containerID="306bf147c5f92c74cfd720cd41ab665a743ccfe77af9159989a0d38672e2c126" exitCode=0
Nov 03 22:05:10.469466 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:10.469061    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b97c67-pl9qn" event={"ID":"b07179fa-cced-46f1-8952-8fe700769c7a","Type":"ContainerDied","Data":"306bf147c5f92c74cfd720cd41ab665a743ccfe77af9159989a0d38672e2c126"}
Nov 03 22:05:10.929782 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:05:10.929718    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:05:11.659652 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:11.659626    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b97c67-pl9qn"
Nov 03 22:05:11.754711 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:11.754622    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b07179fa-cced-46f1-8952-8fe700769c7a-kube-api-access-lvr8g\" (UniqueName: \"kubernetes.io/projected/b07179fa-cced-46f1-8952-8fe700769c7a-kube-api-access-lvr8g\") pod \"b07179fa-cced-46f1-8952-8fe700769c7a\" (UID: \"b07179fa-cced-46f1-8952-8fe700769c7a\") "
Nov 03 22:05:11.754871 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:11.754828    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b07179fa-cced-46f1-8952-8fe700769c7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07179fa-cced-46f1-8952-8fe700769c7a-maas-api-xks-ca\") pod \"b07179fa-cced-46f1-8952-8fe700769c7a\" (UID: \"b07179fa-cced-46f1-8952-8fe700769c7a\") "
Nov 03 22:05:11.754939 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:11.754923    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b07179fa-cced-46f1-8952-8fe700769c7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07179fa-cced-46f1-8952-8fe700769c7a-maas-api-openshift-ca\") pod \"b07179fa-cced-46f1-8952-8fe700769c7a\" (UID: \"b07179fa-cced-46f1-8952-8fe700769c7a\") "
Nov 03 22:05:11.755590 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:11.755284    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b07179fa-cced-46f1-8952-8fe700769c7a-maas-api-openshift-ca" pod "b07179fa-cced-46f1-8952-8fe700769c7a" (UID: "b07179fa-cced-46f1-8952-8fe700769c7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:05:11.755756 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:11.755592    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07179fa-cced-46f1-8952-8fe700769c7a-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:05:11.756994 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:11.756968    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b07179fa-cced-46f1-8952-8fe700769c7a-kube-api-access-lvr8g" pod "b07179fa-cced-46f1-8952-8fe700769c7a" (UID: "b07179fa-cced-46f1-8952-8fe700769c7a"). InnerVolumeSpecName "kube-api-access-lvr8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:05:11.757097 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:11.757081    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b07179fa-cced-46f1-8952-8fe700769c7a-maas-api-xks-ca" pod "b07179fa-cced-46f1-8952-8fe700769c7a" (UID: "b07179fa-cced-46f1-8952-8fe700769c7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:05:11.856688 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:11.856650    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvr8g\" (UniqueName: \"kubernetes.io/projected/b07179fa-cced-46f1-8952-8fe700769c7a-kube-api-access-lvr8g\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:05:11.856688 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:11.856681    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07179fa-cced-46f1-8952-8fe700769c7a-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:05:12.484995 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:12.484961    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b97c67-pl9qn"
Nov 03 22:05:12.485193 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:12.484970    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b97c67-pl9qn" event={"ID":"b07179fa-cced-46f1-8952-8fe700769c7a","Type":"ContainerDied","Data":"b09fc0c6e87a89a572dbf450863da92f69bda9362ad7b3b2f526f754d30729ca"}
Nov 03 22:05:12.485193 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:12.485103    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b09fc0c6e87a89a572dbf450863da92f69bda9362ad7b3b2f526f754d30729ca"
Nov 03 22:05:19.872496 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:19.872455    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac97e0aa-q6dzz"]
Nov 03 22:05:19.873104 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:19.873090    5053 cpu_manager.go:414] "containerMap: removing container" podUID="122b09a6-7732-417d-b82f-a977510c180f" containerName="istio-proxy"
Nov 03 22:05:19.873153 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:19.873105    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="122b09a6-7732-417d-b82f-a977510c180f" containerName="istio-proxy"
Nov 03 22:05:19.873153 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:19.873115    5053 cpu_manager.go:414] "containerMap: removing container" podUID="b07179fa-cced-46f1-8952-8fe700769c7a" containerName="revoke-keys"
Nov 03 22:05:19.873153 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:19.873121    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b07179fa-cced-46f1-8952-8fe700769c7a" containerName="revoke-keys"
Nov 03 22:05:19.873284 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:19.873215    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="b07179fa-cced-46f1-8952-8fe700769c7a" containerName="revoke-keys"
Nov 03 22:05:19.873284 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:19.873227    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="122b09a6-7732-417d-b82f-a977510c180f" containerName="istio-proxy"
Nov 03 22:05:19.876695 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:19.876679    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac97e0aa-q6dzz"
Nov 03 22:05:19.933520 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:19.933487    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac97e0aa-q6dzz"]
Nov 03 22:05:19.944332 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:19.944298    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cf62825-a342-46ec-af65-00b81a746e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac97e0aa-q6dzz\" (UID: \"9cf62825-a342-46ec-af65-00b81a746e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac97e0aa-q6dzz"
Nov 03 22:05:19.944518 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:19.944374    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cf62825-a342-46ec-af65-00b81a746e6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac97e0aa-q6dzz\" (UID: \"9cf62825-a342-46ec-af65-00b81a746e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac97e0aa-q6dzz"
Nov 03 22:05:19.944518 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:19.944438    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgnll\" (UniqueName: \"kubernetes.io/projected/9cf62825-a342-46ec-af65-00b81a746e6a-kube-api-access-hgnll\") pod \"maas-api-revoke-sub-ac97e0aa-q6dzz\" (UID: \"9cf62825-a342-46ec-af65-00b81a746e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac97e0aa-q6dzz"
Nov 03 22:05:19.978013 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:19.977973    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:05:19.978189 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:19.978043    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:05:20.046274 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:20.046161    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cf62825-a342-46ec-af65-00b81a746e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac97e0aa-q6dzz\" (UID: \"9cf62825-a342-46ec-af65-00b81a746e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac97e0aa-q6dzz"
Nov 03 22:05:20.046483 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:20.046328    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cf62825-a342-46ec-af65-00b81a746e6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac97e0aa-q6dzz\" (UID: \"9cf62825-a342-46ec-af65-00b81a746e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac97e0aa-q6dzz"
Nov 03 22:05:20.046483 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:20.046384    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgnll\" (UniqueName: \"kubernetes.io/projected/9cf62825-a342-46ec-af65-00b81a746e6a-kube-api-access-hgnll\") pod \"maas-api-revoke-sub-ac97e0aa-q6dzz\" (UID: \"9cf62825-a342-46ec-af65-00b81a746e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac97e0aa-q6dzz"
Nov 03 22:05:20.046899 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:20.046878    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cf62825-a342-46ec-af65-00b81a746e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac97e0aa-q6dzz\" (UID: \"9cf62825-a342-46ec-af65-00b81a746e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac97e0aa-q6dzz"
Nov 03 22:05:20.048968 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:20.048939    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cf62825-a342-46ec-af65-00b81a746e6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac97e0aa-q6dzz\" (UID: \"9cf62825-a342-46ec-af65-00b81a746e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac97e0aa-q6dzz"
Nov 03 22:05:20.103584 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:20.103551    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgnll\" (UniqueName: \"kubernetes.io/projected/9cf62825-a342-46ec-af65-00b81a746e6a-kube-api-access-hgnll\") pod \"maas-api-revoke-sub-ac97e0aa-q6dzz\" (UID: \"9cf62825-a342-46ec-af65-00b81a746e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac97e0aa-q6dzz"
Nov 03 22:05:20.190473 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:20.190392    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac97e0aa-q6dzz"
Nov 03 22:05:20.571200 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:20.571168    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac97e0aa-q6dzz"]
Nov 03 22:05:20.572152 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:05:20.572130    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf62825_a342_46ec_af65_00b81a746e6a.slice/crio-df757fd19792c19f183bcf22ad0f6d2bdf47fb3bb41cde4c2f688960d1cc2d4e WatchSource:0}: Error finding container df757fd19792c19f183bcf22ad0f6d2bdf47fb3bb41cde4c2f688960d1cc2d4e: Status 404 returned error can't find the container with id df757fd19792c19f183bcf22ad0f6d2bdf47fb3bb41cde4c2f688960d1cc2d4e
Nov 03 22:05:21.550992 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:21.550959    5053 generic.go:356] "Generic (PLEG): container finished" podID="9cf62825-a342-46ec-af65-00b81a746e6a" containerID="adebc406e52a89ef1caf82455df721f175bb2e48cef6226d8aae178b3741a02c" exitCode=0
Nov 03 22:05:21.551416 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:21.551046    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac97e0aa-q6dzz" event={"ID":"9cf62825-a342-46ec-af65-00b81a746e6a","Type":"ContainerDied","Data":"adebc406e52a89ef1caf82455df721f175bb2e48cef6226d8aae178b3741a02c"}
Nov 03 22:05:21.551416 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:21.551102    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac97e0aa-q6dzz" event={"ID":"9cf62825-a342-46ec-af65-00b81a746e6a","Type":"ContainerStarted","Data":"df757fd19792c19f183bcf22ad0f6d2bdf47fb3bb41cde4c2f688960d1cc2d4e"}
Nov 03 22:05:22.780520 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:22.780496    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac97e0aa-q6dzz"
Nov 03 22:05:22.885773 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:22.885670    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cf62825-a342-46ec-af65-00b81a746e6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cf62825-a342-46ec-af65-00b81a746e6a-maas-api-openshift-ca\") pod \"9cf62825-a342-46ec-af65-00b81a746e6a\" (UID: \"9cf62825-a342-46ec-af65-00b81a746e6a\") "
Nov 03 22:05:22.885773 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:22.885743    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cf62825-a342-46ec-af65-00b81a746e6a-kube-api-access-hgnll\" (UniqueName: \"kubernetes.io/projected/9cf62825-a342-46ec-af65-00b81a746e6a-kube-api-access-hgnll\") pod \"9cf62825-a342-46ec-af65-00b81a746e6a\" (UID: \"9cf62825-a342-46ec-af65-00b81a746e6a\") "
Nov 03 22:05:22.886023 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:22.885924    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cf62825-a342-46ec-af65-00b81a746e6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cf62825-a342-46ec-af65-00b81a746e6a-maas-api-xks-ca\") pod \"9cf62825-a342-46ec-af65-00b81a746e6a\" (UID: \"9cf62825-a342-46ec-af65-00b81a746e6a\") "
Nov 03 22:05:22.886082 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:22.886026    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cf62825-a342-46ec-af65-00b81a746e6a-maas-api-openshift-ca" pod "9cf62825-a342-46ec-af65-00b81a746e6a" (UID: "9cf62825-a342-46ec-af65-00b81a746e6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:05:22.886423 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:22.886404    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cf62825-a342-46ec-af65-00b81a746e6a-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:05:22.888058 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:22.888037    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cf62825-a342-46ec-af65-00b81a746e6a-maas-api-xks-ca" pod "9cf62825-a342-46ec-af65-00b81a746e6a" (UID: "9cf62825-a342-46ec-af65-00b81a746e6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:05:22.888058 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:22.888043    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf62825-a342-46ec-af65-00b81a746e6a-kube-api-access-hgnll" pod "9cf62825-a342-46ec-af65-00b81a746e6a" (UID: "9cf62825-a342-46ec-af65-00b81a746e6a"). InnerVolumeSpecName "kube-api-access-hgnll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:05:22.988288 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:22.988214    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cf62825-a342-46ec-af65-00b81a746e6a-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:05:22.988288 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:22.988289    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgnll\" (UniqueName: \"kubernetes.io/projected/9cf62825-a342-46ec-af65-00b81a746e6a-kube-api-access-hgnll\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:05:23.566141 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:23.566109    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac97e0aa-q6dzz"
Nov 03 22:05:23.566372 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:23.566141    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac97e0aa-q6dzz" event={"ID":"9cf62825-a342-46ec-af65-00b81a746e6a","Type":"ContainerDied","Data":"df757fd19792c19f183bcf22ad0f6d2bdf47fb3bb41cde4c2f688960d1cc2d4e"}
Nov 03 22:05:23.566372 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:23.566212    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df757fd19792c19f183bcf22ad0f6d2bdf47fb3bb41cde4c2f688960d1cc2d4e"
Nov 03 22:05:24.046923 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:24.046887    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3112e9fa-68d69"]
Nov 03 22:05:24.049920 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:24.049895    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3112e9fa-68d69"]
Nov 03 22:05:25.667788 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:25.667758    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7e82a8c-058a-467f-870d-361ef4b86532" path="/var/lib/kubelet/pods/a7e82a8c-058a-467f-870d-361ef4b86532/volumes"
Nov 03 22:05:33.234453 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.234416    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"]
Nov 03 22:05:33.235382 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.235360    5053 cpu_manager.go:414] "containerMap: removing container" podUID="9cf62825-a342-46ec-af65-00b81a746e6a" containerName="revoke-keys"
Nov 03 22:05:33.235441 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.235386    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cf62825-a342-46ec-af65-00b81a746e6a" containerName="revoke-keys"
Nov 03 22:05:33.235598 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.235575    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cf62825-a342-46ec-af65-00b81a746e6a" containerName="revoke-keys"
Nov 03 22:05:33.241052 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.241028    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.293111 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.293083    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-27a3c3-openshift-default-dockercfg-srpbn\""
Nov 03 22:05:33.328556 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.328507    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-istio-envoy\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.328783 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.328568    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-workload-socket\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.328783 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.328752    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-credential-socket\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.328892 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.328842    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af213d85-c84b-4ae2-b533-77c12c615a96-istio-token\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.328892 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.328875    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-workload-certs\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.328986 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.328907    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af213d85-c84b-4ae2-b533-77c12c615a96-istio-podinfo\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.328986 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.328949    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4bhd\" (UniqueName: \"kubernetes.io/projected/af213d85-c84b-4ae2-b533-77c12c615a96-kube-api-access-g4bhd\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.329156 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.329135    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-istio-data\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.329220 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.329184    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af213d85-c84b-4ae2-b533-77c12c615a96-istiod-ca-cert\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.348895 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.348860    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"]
Nov 03 22:05:33.430763 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.430725    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-credential-socket\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.430974 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.430813    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af213d85-c84b-4ae2-b533-77c12c615a96-istio-token\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.430974 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.430850    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-workload-certs\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.430974 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.430879    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af213d85-c84b-4ae2-b533-77c12c615a96-istio-podinfo\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.430974 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.430920    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4bhd\" (UniqueName: \"kubernetes.io/projected/af213d85-c84b-4ae2-b533-77c12c615a96-kube-api-access-g4bhd\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.431226 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.431055    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-istio-data\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.431226 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.431092    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af213d85-c84b-4ae2-b533-77c12c615a96-istiod-ca-cert\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.431226 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.431129    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-istio-envoy\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.431226 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.431166    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-workload-socket\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.431226 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.431186    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-credential-socket\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.431629 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.431227    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-workload-certs\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.431629 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.431421    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-istio-data\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.431629 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.431568    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-workload-socket\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.431812 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.431793    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af213d85-c84b-4ae2-b533-77c12c615a96-istiod-ca-cert\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.433909 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.433888    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af213d85-c84b-4ae2-b533-77c12c615a96-istio-podinfo\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.435010 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.434978    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-istio-envoy\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.488463 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.488396    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4bhd\" (UniqueName: \"kubernetes.io/projected/af213d85-c84b-4ae2-b533-77c12c615a96-kube-api-access-g4bhd\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.488463 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.488408    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af213d85-c84b-4ae2-b533-77c12c615a96-istio-token\") pod \"e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") " pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.557582 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.557550    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:33.652810 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.652759    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"]
Nov 03 22:05:33.658271 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.658230    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:05:33.710780 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.710743    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-87e1cf\"/\"kube-root-ca.crt\""
Nov 03 22:05:33.710954 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.710860    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-87e1cf\"/\"default-dockercfg-77mh2\""
Nov 03 22:05:33.710954 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.710940    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-87e1cf\"/\"auth-test-model-87e1cf-kserve-self-signed-certs\""
Nov 03 22:05:33.711091 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.711007    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-87e1cf\"/\"openshift-service-ca.crt\""
Nov 03 22:05:33.712846 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.712812    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"]
Nov 03 22:05:33.846769 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.846688    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-dshm\") pod \"auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd\" (UID: \"225eed60-7bae-455a-baac-3c013d12a51c\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:05:33.846769 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.846740    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/225eed60-7bae-455a-baac-3c013d12a51c-tls-certs\") pod \"auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd\" (UID: \"225eed60-7bae-455a-baac-3c013d12a51c\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:05:33.847027 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.846783    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-tmp-dir\") pod \"auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd\" (UID: \"225eed60-7bae-455a-baac-3c013d12a51c\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:05:33.847027 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.846958    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-model-cache\") pod \"auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd\" (UID: \"225eed60-7bae-455a-baac-3c013d12a51c\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:05:33.847145 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.847048    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-home\") pod \"auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd\" (UID: \"225eed60-7bae-455a-baac-3c013d12a51c\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:05:33.847323 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.847303    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkrrk\" (UniqueName: \"kubernetes.io/projected/225eed60-7bae-455a-baac-3c013d12a51c-kube-api-access-xkrrk\") pod \"auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd\" (UID: \"225eed60-7bae-455a-baac-3c013d12a51c\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:05:33.948464 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.948426    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-tmp-dir\") pod \"auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd\" (UID: \"225eed60-7bae-455a-baac-3c013d12a51c\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:05:33.948674 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.948490    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-model-cache\") pod \"auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd\" (UID: \"225eed60-7bae-455a-baac-3c013d12a51c\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:05:33.948674 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.948526    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-home\") pod \"auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd\" (UID: \"225eed60-7bae-455a-baac-3c013d12a51c\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:05:33.948674 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.948668    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkrrk\" (UniqueName: \"kubernetes.io/projected/225eed60-7bae-455a-baac-3c013d12a51c-kube-api-access-xkrrk\") pod \"auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd\" (UID: \"225eed60-7bae-455a-baac-3c013d12a51c\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:05:33.948869 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.948727    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-dshm\") pod \"auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd\" (UID: \"225eed60-7bae-455a-baac-3c013d12a51c\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:05:33.948869 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.948769    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/225eed60-7bae-455a-baac-3c013d12a51c-tls-certs\") pod \"auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd\" (UID: \"225eed60-7bae-455a-baac-3c013d12a51c\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:05:33.948869 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.948851    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-tmp-dir\") pod \"auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd\" (UID: \"225eed60-7bae-455a-baac-3c013d12a51c\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:05:33.949024 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.948901    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-model-cache\") pod \"auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd\" (UID: \"225eed60-7bae-455a-baac-3c013d12a51c\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:05:33.949024 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.948916    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-home\") pod \"auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd\" (UID: \"225eed60-7bae-455a-baac-3c013d12a51c\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:05:33.951193 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.951174    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-dshm\") pod \"auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd\" (UID: \"225eed60-7bae-455a-baac-3c013d12a51c\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:05:33.951485 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:33.951469    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/225eed60-7bae-455a-baac-3c013d12a51c-tls-certs\") pod \"auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd\" (UID: \"225eed60-7bae-455a-baac-3c013d12a51c\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:05:34.007924 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:34.007896    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkrrk\" (UniqueName: \"kubernetes.io/projected/225eed60-7bae-455a-baac-3c013d12a51c-kube-api-access-xkrrk\") pod \"auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd\" (UID: \"225eed60-7bae-455a-baac-3c013d12a51c\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:05:34.008726 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:34.008699    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"]
Nov 03 22:05:34.009084 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:05:34.009056    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf213d85_c84b_4ae2_b533_77c12c615a96.slice/crio-e4f04727252ca25b6129f5de7db67830b39f860cd9a41749a7df29b5dc3a6ba8 WatchSource:0}: Error finding container e4f04727252ca25b6129f5de7db67830b39f860cd9a41749a7df29b5dc3a6ba8: Status 404 returned error can't find the container with id e4f04727252ca25b6129f5de7db67830b39f860cd9a41749a7df29b5dc3a6ba8
Nov 03 22:05:34.011783 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:34.011747    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:05:34.011865 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:34.011816    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:05:34.011865 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:34.011853    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:05:34.278286 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:34.278254    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:05:34.654989 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:34.654961    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"]
Nov 03 22:05:34.656604 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:05:34.656578    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod225eed60_7bae_455a_baac_3c013d12a51c.slice/crio-d610bfc5ee901918b9ec612ef4b2842306bb6ed0679f21fa9f55679e80610f30 WatchSource:0}: Error finding container d610bfc5ee901918b9ec612ef4b2842306bb6ed0679f21fa9f55679e80610f30: Status 404 returned error can't find the container with id d610bfc5ee901918b9ec612ef4b2842306bb6ed0679f21fa9f55679e80610f30
Nov 03 22:05:34.663941 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:34.663912    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv" event={"ID":"af213d85-c84b-4ae2-b533-77c12c615a96","Type":"ContainerStarted","Data":"76e2a573d15add655d1d5f6c3908d836cb930d51fd03c35ceef81f90e4024f10"}
Nov 03 22:05:34.664077 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:34.663969    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv" event={"ID":"af213d85-c84b-4ae2-b533-77c12c615a96","Type":"ContainerStarted","Data":"e4f04727252ca25b6129f5de7db67830b39f860cd9a41749a7df29b5dc3a6ba8"}
Nov 03 22:05:34.783964 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:34.783915    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv" podStartSLOduration=1.7838976359999998 podStartE2EDuration="1.783897636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:05:34.778945387 +0000 UTC m=+3049.272867262" watchObservedRunningTime="2026-11-03 22:05:34.783897636 +0000 UTC m=+3049.277819511"
Nov 03 22:05:35.557705 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:35.557667    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:35.563523 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:35.563498    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:35.673459 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:35.673421    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd" event={"ID":"225eed60-7bae-455a-baac-3c013d12a51c","Type":"ContainerStarted","Data":"0c714f7ac0ee0f79ea2ac9746ec9a2289075f5d19db5e0cabf657d74b0abd7a0"}
Nov 03 22:05:35.673459 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:35.673464    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd" event={"ID":"225eed60-7bae-455a-baac-3c013d12a51c","Type":"ContainerStarted","Data":"d610bfc5ee901918b9ec612ef4b2842306bb6ed0679f21fa9f55679e80610f30"}
Nov 03 22:05:35.673733 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:35.673667    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:35.674893 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:35.674870    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:05:35.917000 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:35.916903    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd" podStartSLOduration=2.916889815 podStartE2EDuration="2.916889815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:05:35.913610712 +0000 UTC m=+3050.407532588" watchObservedRunningTime="2026-11-03 22:05:35.916889815 +0000 UTC m=+3050.410811691"
Nov 03 22:05:36.279370 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:36.279334    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:05:40.940692 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:05:40.940645    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:05:44.279187 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:44.279148    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:05:44.292060 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:44.292035    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:05:44.750971 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:44.750943    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:05:46.889700 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:46.889661    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"]
Nov 03 22:05:46.896270 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:46.895986    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"
Nov 03 22:05:46.948268 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:46.947838    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"]
Nov 03 22:05:46.948477 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:46.948309    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bef7697e-1c56-48a7-9758-3243cd62e8f1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv\" (UID: \"bef7697e-1c56-48a7-9758-3243cd62e8f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"
Nov 03 22:05:46.948555 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:46.948480    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef7697e-1c56-48a7-9758-3243cd62e8f1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv\" (UID: \"bef7697e-1c56-48a7-9758-3243cd62e8f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"
Nov 03 22:05:46.948555 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:46.948531    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5s9f\" (UniqueName: \"kubernetes.io/projected/bef7697e-1c56-48a7-9758-3243cd62e8f1-kube-api-access-n5s9f\") pod \"maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv\" (UID: \"bef7697e-1c56-48a7-9758-3243cd62e8f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"
Nov 03 22:05:46.948660 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:46.948600    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bef7697e-1c56-48a7-9758-3243cd62e8f1-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv\" (UID: \"bef7697e-1c56-48a7-9758-3243cd62e8f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"
Nov 03 22:05:46.992508 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:46.992475    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-0d45ba-serving-cert\""
Nov 03 22:05:47.050088 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:47.050003    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bef7697e-1c56-48a7-9758-3243cd62e8f1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv\" (UID: \"bef7697e-1c56-48a7-9758-3243cd62e8f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"
Nov 03 22:05:47.052305 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:47.051285    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef7697e-1c56-48a7-9758-3243cd62e8f1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv\" (UID: \"bef7697e-1c56-48a7-9758-3243cd62e8f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"
Nov 03 22:05:47.052549 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:05:47.051050    5053 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-serving-cert: secret "maas-api-e2e-shared-b-w4-0d45ba-serving-cert" not found
Nov 03 22:05:47.052713 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:05:47.052699    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef7697e-1c56-48a7-9758-3243cd62e8f1-maas-api-tls podName:bef7697e-1c56-48a7-9758-3243cd62e8f1 nodeName:}" failed. No retries permitted until 2026-11-03 22:05:47.552676093 +0000 UTC m=+3062.046597959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bef7697e-1c56-48a7-9758-3243cd62e8f1-maas-api-tls") pod "maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv" (UID: "bef7697e-1c56-48a7-9758-3243cd62e8f1") : secret "maas-api-e2e-shared-b-w4-0d45ba-serving-cert" not found
Nov 03 22:05:47.052947 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:47.052547    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef7697e-1c56-48a7-9758-3243cd62e8f1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv\" (UID: \"bef7697e-1c56-48a7-9758-3243cd62e8f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"
Nov 03 22:05:47.053133 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:47.053095    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5s9f\" (UniqueName: \"kubernetes.io/projected/bef7697e-1c56-48a7-9758-3243cd62e8f1-kube-api-access-n5s9f\") pod \"maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv\" (UID: \"bef7697e-1c56-48a7-9758-3243cd62e8f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"
Nov 03 22:05:47.053290 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:47.053171    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bef7697e-1c56-48a7-9758-3243cd62e8f1-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv\" (UID: \"bef7697e-1c56-48a7-9758-3243cd62e8f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"
Nov 03 22:05:47.062840 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:47.062809    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bef7697e-1c56-48a7-9758-3243cd62e8f1-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv\" (UID: \"bef7697e-1c56-48a7-9758-3243cd62e8f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"
Nov 03 22:05:47.110898 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:47.110862    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5s9f\" (UniqueName: \"kubernetes.io/projected/bef7697e-1c56-48a7-9758-3243cd62e8f1-kube-api-access-n5s9f\") pod \"maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv\" (UID: \"bef7697e-1c56-48a7-9758-3243cd62e8f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"
Nov 03 22:05:47.560186 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:47.560153    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bef7697e-1c56-48a7-9758-3243cd62e8f1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv\" (UID: \"bef7697e-1c56-48a7-9758-3243cd62e8f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"
Nov 03 22:05:47.562769 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:47.562745    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bef7697e-1c56-48a7-9758-3243cd62e8f1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv\" (UID: \"bef7697e-1c56-48a7-9758-3243cd62e8f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"
Nov 03 22:05:47.815969 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:47.815862    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"
Nov 03 22:05:47.827055 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:47.827026    5053 scope.go:122] "RemoveContainer" containerID="747b2dcacce878c5ac6a4d24e4a08e489e1dadd5e0d7f0492be4923ae0ff506c"
Nov 03 22:05:47.861720 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:47.861691    5053 scope.go:122] "RemoveContainer" containerID="82a2f80ea847d4ab602e0576ee1cbfe3a0fc051474415a8d061d841a1cbb670b"
Nov 03 22:05:48.200877 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:48.200839    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"]
Nov 03 22:05:48.202040 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:05:48.202002    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef7697e_1c56_48a7_9758_3243cd62e8f1.slice/crio-87de27ba7b54c033200147704433e6dfcce45d97972d601038a98916bb9785ad WatchSource:0}: Error finding container 87de27ba7b54c033200147704433e6dfcce45d97972d601038a98916bb9785ad: Status 404 returned error can't find the container with id 87de27ba7b54c033200147704433e6dfcce45d97972d601038a98916bb9785ad
Nov 03 22:05:48.771433 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:48.771401    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv" event={"ID":"bef7697e-1c56-48a7-9758-3243cd62e8f1","Type":"ContainerStarted","Data":"87de27ba7b54c033200147704433e6dfcce45d97972d601038a98916bb9785ad"}
Nov 03 22:05:49.779883 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:49.779849    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv" event={"ID":"bef7697e-1c56-48a7-9758-3243cd62e8f1","Type":"ContainerStarted","Data":"a5232b3ad663f517c8b08eddca19b9353d745adf7868452652f6e940cabab80a"}
Nov 03 22:05:49.780323 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:49.780065    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"
Nov 03 22:05:55.793562 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:55.793534    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"
Nov 03 22:05:55.908845 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:05:55.908782    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv" podStartSLOduration=9.063440692 podStartE2EDuration="9.908766979s" totalImagesPullingTime="845.326287ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:05:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:05:49.897524408 +0000 UTC m=+3064.391446284" watchObservedRunningTime="2026-11-03 22:05:55.908766979 +0000 UTC m=+3070.402688881"
Nov 03 22:06:06.199186 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:06.199150    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"]
Nov 03 22:06:06.204998 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:06.204973    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"
Nov 03 22:06:06.237791 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:06.237755    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb\" (UID: \"47fec592-cd7c-4208-92e9-c8dc38e1a7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"
Nov 03 22:06:06.237978 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:06.237958    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb\" (UID: \"47fec592-cd7c-4208-92e9-c8dc38e1a7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"
Nov 03 22:06:06.238133 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:06.238114    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb\" (UID: \"47fec592-cd7c-4208-92e9-c8dc38e1a7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"
Nov 03 22:06:06.238186 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:06.238171    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88v7w\" (UniqueName: \"kubernetes.io/projected/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-kube-api-access-88v7w\") pod \"maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb\" (UID: \"47fec592-cd7c-4208-92e9-c8dc38e1a7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"
Nov 03 22:06:06.261554 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:06.261517    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"]
Nov 03 22:06:06.305855 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:06.305802    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-75f577-serving-cert\""
Nov 03 22:06:06.339589 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:06.339552    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb\" (UID: \"47fec592-cd7c-4208-92e9-c8dc38e1a7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"
Nov 03 22:06:06.339792 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:06.339712    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb\" (UID: \"47fec592-cd7c-4208-92e9-c8dc38e1a7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"
Nov 03 22:06:06.339792 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:06.339766    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88v7w\" (UniqueName: \"kubernetes.io/projected/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-kube-api-access-88v7w\") pod \"maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb\" (UID: \"47fec592-cd7c-4208-92e9-c8dc38e1a7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"
Nov 03 22:06:06.339939 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:06.339835    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb\" (UID: \"47fec592-cd7c-4208-92e9-c8dc38e1a7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"
Nov 03 22:06:06.340080 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:06:06.340054    5053 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-serving-cert: secret "maas-api-e2e-shared-b-w0-75f577-serving-cert" not found
Nov 03 22:06:06.340163 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:06:06.340148    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-maas-api-tls podName:47fec592-cd7c-4208-92e9-c8dc38e1a7f1 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:06.840126562 +0000 UTC m=+3081.334048435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-maas-api-tls") pod "maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb" (UID: "47fec592-cd7c-4208-92e9-c8dc38e1a7f1") : secret "maas-api-e2e-shared-b-w0-75f577-serving-cert" not found
Nov 03 22:06:06.340684 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:06.340658    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb\" (UID: \"47fec592-cd7c-4208-92e9-c8dc38e1a7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"
Nov 03 22:06:06.349304 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:06.342695    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb\" (UID: \"47fec592-cd7c-4208-92e9-c8dc38e1a7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"
Nov 03 22:06:06.399184 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:06.399154    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88v7w\" (UniqueName: \"kubernetes.io/projected/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-kube-api-access-88v7w\") pod \"maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb\" (UID: \"47fec592-cd7c-4208-92e9-c8dc38e1a7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"
Nov 03 22:06:06.849448 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:06.849398    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb\" (UID: \"47fec592-cd7c-4208-92e9-c8dc38e1a7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"
Nov 03 22:06:06.852210 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:06.852188    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb\" (UID: \"47fec592-cd7c-4208-92e9-c8dc38e1a7f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"
Nov 03 22:06:07.121015 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:07.120923    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"
Nov 03 22:06:07.566650 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:07.536713    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"]
Nov 03 22:06:07.935585 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:07.935493    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb" event={"ID":"47fec592-cd7c-4208-92e9-c8dc38e1a7f1","Type":"ContainerStarted","Data":"fd9e59962f0423efb12dac98d4cf9ea87de109fb9389480c837f54b23096fd3a"}
Nov 03 22:06:08.944166 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:08.944130    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb" event={"ID":"47fec592-cd7c-4208-92e9-c8dc38e1a7f1","Type":"ContainerStarted","Data":"78e6e4194eabfb434203f347007c2159b87bfff2b3a9ffeab6726b386a462c2c"}
Nov 03 22:06:08.957572 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:08.944223    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"
Nov 03 22:06:10.932266 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:06:10.932207    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:06:14.957230 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:14.957200    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"
Nov 03 22:06:15.072306 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:15.072255    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb" podStartSLOduration=8.146599573 podStartE2EDuration="9.072226355s" totalImagesPullingTime="925.626782ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:06:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:09.089924899 +0000 UTC m=+3083.583846773" watchObservedRunningTime="2026-11-03 22:06:15.072226355 +0000 UTC m=+3089.566148229"
Nov 03 22:06:40.930167 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:06:40.930127    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:06:41.286737 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:41.286706    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"]
Nov 03 22:06:41.292141 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:41.292116    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"
Nov 03 22:06:41.348708 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:41.348672    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"]
Nov 03 22:06:41.392565 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:41.392531    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-27a3c3-serving-cert\""
Nov 03 22:06:41.462321 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:41.462279    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-maas-api-tls\") pod \"maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn\" (UID: \"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"
Nov 03 22:06:41.462522 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:41.462393    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-metrics-certs\") pod \"maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn\" (UID: \"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"
Nov 03 22:06:41.462522 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:41.462476    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-openshift-service-ca\") pod \"maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn\" (UID: \"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"
Nov 03 22:06:41.462664 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:41.462576    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr2fl\" (UniqueName: \"kubernetes.io/projected/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-kube-api-access-rr2fl\") pod \"maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn\" (UID: \"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"
Nov 03 22:06:41.563900 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:41.563801    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-metrics-certs\") pod \"maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn\" (UID: \"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"
Nov 03 22:06:41.564072 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:41.563962    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-openshift-service-ca\") pod \"maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn\" (UID: \"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"
Nov 03 22:06:41.564127 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:41.564113    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr2fl\" (UniqueName: \"kubernetes.io/projected/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-kube-api-access-rr2fl\") pod \"maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn\" (UID: \"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"
Nov 03 22:06:41.564294 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:41.564272    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-maas-api-tls\") pod \"maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn\" (UID: \"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"
Nov 03 22:06:41.564544 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:06:41.564512    5053 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-serving-cert: secret "maas-api-e2e-explicit-27a3c3-serving-cert" not found
Nov 03 22:06:41.564668 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:06:41.564618    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-maas-api-tls podName:ce0ed29f-9f1e-4ac2-80d5-b806fa23a225 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:42.0645961 +0000 UTC m=+3116.558517964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-maas-api-tls") pod "maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn" (UID: "ce0ed29f-9f1e-4ac2-80d5-b806fa23a225") : secret "maas-api-e2e-explicit-27a3c3-serving-cert" not found
Nov 03 22:06:41.565205 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:41.565171    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-openshift-service-ca\") pod \"maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn\" (UID: \"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"
Nov 03 22:06:41.567763 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:41.567736    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-metrics-certs\") pod \"maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn\" (UID: \"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"
Nov 03 22:06:41.623858 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:41.623820    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr2fl\" (UniqueName: \"kubernetes.io/projected/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-kube-api-access-rr2fl\") pod \"maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn\" (UID: \"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"
Nov 03 22:06:42.070809 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:42.070770    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-maas-api-tls\") pod \"maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn\" (UID: \"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"
Nov 03 22:06:42.073484 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:42.073462    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-maas-api-tls\") pod \"maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn\" (UID: \"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"
Nov 03 22:06:42.207458 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:42.207416    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"
Nov 03 22:06:42.590414 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:42.590374    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"]
Nov 03 22:06:42.590615 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:06:42.590591    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0ed29f_9f1e_4ac2_80d5_b806fa23a225.slice/crio-4294b0d3503a845c7972706f7e5ed5f1b0bb947bcb277eeca6ea749519f4f25e WatchSource:0}: Error finding container 4294b0d3503a845c7972706f7e5ed5f1b0bb947bcb277eeca6ea749519f4f25e: Status 404 returned error can't find the container with id 4294b0d3503a845c7972706f7e5ed5f1b0bb947bcb277eeca6ea749519f4f25e
Nov 03 22:06:43.193137 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:43.193097    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn" event={"ID":"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225","Type":"ContainerStarted","Data":"4294b0d3503a845c7972706f7e5ed5f1b0bb947bcb277eeca6ea749519f4f25e"}
Nov 03 22:06:44.201252 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:44.201204    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn" event={"ID":"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225","Type":"ContainerStarted","Data":"ede53729f16145eeebcb079a9f48bc0d716a5a2d9170a43615c543c8de74a9fc"}
Nov 03 22:06:44.202074 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:44.201412    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"
Nov 03 22:06:47.918155 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:47.918123    5053 scope.go:122] "RemoveContainer" containerID="0d1f620d4e229a01fc761c02d028d1206ef841079ad91b74b652df8283e61bff"
Nov 03 22:06:50.212976 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:50.212947    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"
Nov 03 22:06:50.327391 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:06:50.327339    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn" podStartSLOduration=8.551735755 podStartE2EDuration="9.327325205s" totalImagesPullingTime="775.58945ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:06:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:44.343378325 +0000 UTC m=+3118.837300206" watchObservedRunningTime="2026-11-03 22:06:50.327325205 +0000 UTC m=+3124.821247079"
Nov 03 22:07:08.335282 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:08.335173    5053 patch_prober.go:29] interesting pod/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:07:08.335721 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:08.335275    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf" podUID="0c981726-d50f-4694-93d4-ebefb45bfb47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:07:10.415565 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:10.415518    5053 generic.go:356] "Generic (PLEG): container finished" podID="0c981726-d50f-4694-93d4-ebefb45bfb47" containerID="0c20c4d038d8d2aba7c4324a914c750bb2fd8aa17bcea5c98b7140aafe24eb51" exitCode=0
Nov 03 22:07:10.416291 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:10.415587    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf" event={"ID":"0c981726-d50f-4694-93d4-ebefb45bfb47","Type":"ContainerDied","Data":"0c20c4d038d8d2aba7c4324a914c750bb2fd8aa17bcea5c98b7140aafe24eb51"}
Nov 03 22:07:10.416291 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:10.416014    5053 scope.go:122] "RemoveContainer" containerID="0c20c4d038d8d2aba7c4324a914c750bb2fd8aa17bcea5c98b7140aafe24eb51"
Nov 03 22:07:10.417932 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:10.417901    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:07:10.418032 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:10.417972    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:07:10.418032 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:10.418005    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:07:10.931037 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:07:10.930994    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:07:11.434877 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:11.434839    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf" event={"ID":"0c981726-d50f-4694-93d4-ebefb45bfb47","Type":"ContainerStarted","Data":"c35dc0340a1c39f0945da7cd44caa61e92c613acae1761c117c9c45278b8a5dc"}
Nov 03 22:07:12.106976 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:12.106888    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:07:12.107380 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:12.107357    5053 patch_prober.go:29] interesting pod/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 03 22:07:12.107567 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:12.107543    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf" podUID="0c981726-d50f-4694-93d4-ebefb45bfb47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 03 22:07:12.443170 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:12.443138    5053 generic.go:356] "Generic (PLEG): container finished" podID="af213d85-c84b-4ae2-b533-77c12c615a96" containerID="76e2a573d15add655d1d5f6c3908d836cb930d51fd03c35ceef81f90e4024f10" exitCode=0
Nov 03 22:07:12.443621 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:12.443213    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv" event={"ID":"af213d85-c84b-4ae2-b533-77c12c615a96","Type":"ContainerDied","Data":"76e2a573d15add655d1d5f6c3908d836cb930d51fd03c35ceef81f90e4024f10"}
Nov 03 22:07:12.443691 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:12.443621    5053 scope.go:122] "RemoveContainer" containerID="76e2a573d15add655d1d5f6c3908d836cb930d51fd03c35ceef81f90e4024f10"
Nov 03 22:07:12.445116 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:12.445085    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:07:12.445257 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:12.445150    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:07:12.445257 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:12.445184    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:07:13.106678 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:13.106637    5053 patch_prober.go:29] interesting pod/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 03 22:07:13.106975 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:13.106944    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf" podUID="0c981726-d50f-4694-93d4-ebefb45bfb47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 03 22:07:13.455338 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:13.455294    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv" event={"ID":"af213d85-c84b-4ae2-b533-77c12c615a96","Type":"ContainerStarted","Data":"3e36aebae6007cd474d5192428752a149de8140293faf0d7b6c918af509db9fd"}
Nov 03 22:07:13.558599 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:13.558560    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:07:13.558805 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:13.558721    5053 patch_prober.go:29] interesting pod/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Nov 03 22:07:13.558805 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:13.558779    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv" podUID="af213d85-c84b-4ae2-b533-77c12c615a96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Nov 03 22:07:14.109271 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:14.106614    5053 patch_prober.go:29] interesting pod/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 03 22:07:14.109271 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:14.106677    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf" podUID="0c981726-d50f-4694-93d4-ebefb45bfb47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 03 22:07:14.562297 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:14.561679    5053 patch_prober.go:29] interesting pod/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Nov 03 22:07:14.562297 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:14.561739    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv" podUID="af213d85-c84b-4ae2-b533-77c12c615a96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Nov 03 22:07:15.106273 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:15.104619    5053 patch_prober.go:29] interesting pod/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 03 22:07:15.106273 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:15.104676    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf" podUID="0c981726-d50f-4694-93d4-ebefb45bfb47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 03 22:07:15.562824 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:15.562785    5053 patch_prober.go:29] interesting pod/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Nov 03 22:07:15.563488 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:15.563453    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv" podUID="af213d85-c84b-4ae2-b533-77c12c615a96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Nov 03 22:07:16.119263 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:16.118642    5053 patch_prober.go:29] interesting pod/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 03 22:07:16.119263 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:16.118702    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf" podUID="0c981726-d50f-4694-93d4-ebefb45bfb47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 03 22:07:16.558715 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:16.558672    5053 patch_prober.go:29] interesting pod/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Nov 03 22:07:16.558941 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:16.558748    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv" podUID="af213d85-c84b-4ae2-b533-77c12c615a96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Nov 03 22:07:17.118167 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:17.116522    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:07:17.118167 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:17.118005    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:07:17.126500 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:17.126445    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:07:17.558387 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:17.558357    5053 patch_prober.go:29] interesting pod/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Nov 03 22:07:17.558593 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:17.558415    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv" podUID="af213d85-c84b-4ae2-b533-77c12c615a96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Nov 03 22:07:18.558024 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:18.557989    5053 patch_prober.go:29] interesting pod/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Nov 03 22:07:18.558573 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:18.558058    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv" podUID="af213d85-c84b-4ae2-b533-77c12c615a96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Nov 03 22:07:19.563570 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:19.563541    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:07:19.564042 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:19.564009    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:07:19.564764 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:19.564743    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:07:20.414861 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:20.414827    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"]
Nov 03 22:07:20.419388 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:20.419365    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"
Nov 03 22:07:20.472523 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:20.472487    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-7fb477-serving-cert\""
Nov 03 22:07:20.475595 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:20.475566    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"]
Nov 03 22:07:20.524807 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:20.524777    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlx79\" (UniqueName: \"kubernetes.io/projected/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-kube-api-access-tlx79\") pod \"maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf\" (UID: \"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"
Nov 03 22:07:20.525001 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:20.524901    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf\" (UID: \"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"
Nov 03 22:07:20.525044 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:20.525019    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf\" (UID: \"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"
Nov 03 22:07:20.525080 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:20.525065    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf\" (UID: \"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"
Nov 03 22:07:20.626846 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:20.626814    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf\" (UID: \"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"
Nov 03 22:07:20.627343 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:20.626880    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf\" (UID: \"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"
Nov 03 22:07:20.627343 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:07:20.626999    5053 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-serving-cert: secret "maas-api-e2e-shared-b-w3-7fb477-serving-cert" not found
Nov 03 22:07:20.627343 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:20.627015    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlx79\" (UniqueName: \"kubernetes.io/projected/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-kube-api-access-tlx79\") pod \"maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf\" (UID: \"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"
Nov 03 22:07:20.627343 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:07:20.627122    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-maas-api-tls podName:bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:21.127087138 +0000 UTC m=+3155.621009011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-maas-api-tls") pod "maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf" (UID: "bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518") : secret "maas-api-e2e-shared-b-w3-7fb477-serving-cert" not found
Nov 03 22:07:20.627343 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:20.627219    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf\" (UID: \"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"
Nov 03 22:07:20.628152 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:20.628129    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf\" (UID: \"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"
Nov 03 22:07:20.630314 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:20.630292    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf\" (UID: \"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"
Nov 03 22:07:20.688965 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:20.687683    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlx79\" (UniqueName: \"kubernetes.io/projected/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-kube-api-access-tlx79\") pod \"maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf\" (UID: \"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"
Nov 03 22:07:21.135776 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:21.135732    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf\" (UID: \"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"
Nov 03 22:07:21.138440 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:21.138409    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf\" (UID: \"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"
Nov 03 22:07:21.333510 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:21.333476    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"
Nov 03 22:07:21.711705 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:21.711674    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"]
Nov 03 22:07:21.712770 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:07:21.712744    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbd93ef_5bd8_4fdf_aecb_c425fe7a5518.slice/crio-a42e8bd5e72e1ea5d12440a7ce56a3c8bfa9f0fce23c53b6bc0a0fd5bc4b9e4a WatchSource:0}: Error finding container a42e8bd5e72e1ea5d12440a7ce56a3c8bfa9f0fce23c53b6bc0a0fd5bc4b9e4a: Status 404 returned error can't find the container with id a42e8bd5e72e1ea5d12440a7ce56a3c8bfa9f0fce23c53b6bc0a0fd5bc4b9e4a
Nov 03 22:07:22.526812 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:22.526770    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf" event={"ID":"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518","Type":"ContainerStarted","Data":"a42e8bd5e72e1ea5d12440a7ce56a3c8bfa9f0fce23c53b6bc0a0fd5bc4b9e4a"}
Nov 03 22:07:23.535520 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:23.535482    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf" event={"ID":"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518","Type":"ContainerStarted","Data":"e93a792cd716e63ad9e231e80debee828f7d8e219328b48c3daed07f13e63bf7"}
Nov 03 22:07:23.535974 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:23.535551    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"
Nov 03 22:07:23.657046 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:23.656981    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf" podStartSLOduration=2.808363123 podStartE2EDuration="3.656958763s" totalImagesPullingTime="848.59564ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:23.650706014 +0000 UTC m=+3158.144627890" watchObservedRunningTime="2026-11-03 22:07:23.656958763 +0000 UTC m=+3158.150880645"
Nov 03 22:07:29.563258 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:29.555136    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"
Nov 03 22:07:40.930285 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:07:40.930205    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:07:48.743212 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:48.743170    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"]
Nov 03 22:07:48.743922 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:48.743438    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx" podUID="5fb7d89c-af78-4e10-bfe9-ccbbb51e9222" containerName="istio-proxy" containerID="cri-o://46a20bb03cfa578f8967ea82fcf7b0154b772d24ce057acac6e3ac9f4d8a60ea" gracePeriod=30
Nov 03 22:07:48.795458 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:48.795418    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"]
Nov 03 22:07:48.795780 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:48.795747    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh" podUID="0626b947-6a73-4669-b426-b5b9ce1c30e3" containerName="istio-proxy" containerID="cri-o://3e1f6328dcc2a3a03c4cdbae3a79687526a8864535f3ca532cc932acae33829c" gracePeriod=30
Nov 03 22:07:52.046216 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:52.046175    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1324f3a4-lz8vw"]
Nov 03 22:07:52.095201 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:52.095163    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1324f3a4-lz8vw"]
Nov 03 22:07:53.669598 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:53.669565    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a601e88-a602-4478-a0c7-2d1e07852968" path="/var/lib/kubelet/pods/0a601e88-a602-4478-a0c7-2d1e07852968/volumes"
Nov 03 22:07:54.129757 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.129726    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:07:54.178812 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.178747    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:07:54.213380 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.213347    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-workload-socket\") pod \"0626b947-6a73-4669-b426-b5b9ce1c30e3\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") "
Nov 03 22:07:54.213652 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.213389    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-credential-socket\") pod \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") "
Nov 03 22:07:54.213652 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.213425    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-credential-socket\") pod \"0626b947-6a73-4669-b426-b5b9ce1c30e3\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") "
Nov 03 22:07:54.213652 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.213481    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-podinfo\") pod \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") "
Nov 03 22:07:54.213652 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.213544    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istiod-ca-cert\") pod \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") "
Nov 03 22:07:54.213652 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.213574    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-data\") pod \"0626b947-6a73-4669-b426-b5b9ce1c30e3\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") "
Nov 03 22:07:54.213652 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.213618    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-token\" (UniqueName: \"kubernetes.io/projected/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-token\") pod \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") "
Nov 03 22:07:54.213652 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.213649    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-kube-api-access-pkbd7\" (UniqueName: \"kubernetes.io/projected/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-kube-api-access-pkbd7\") pod \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") "
Nov 03 22:07:54.213985 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.213684    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0626b947-6a73-4669-b426-b5b9ce1c30e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0626b947-6a73-4669-b426-b5b9ce1c30e3-istiod-ca-cert\") pod \"0626b947-6a73-4669-b426-b5b9ce1c30e3\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") "
Nov 03 22:07:54.213985 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.213705    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-credential-socket" pod "5fb7d89c-af78-4e10-bfe9-ccbbb51e9222" (UID: "5fb7d89c-af78-4e10-bfe9-ccbbb51e9222"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:07:54.213985 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.213714    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-data\") pod \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") "
Nov 03 22:07:54.213985 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.213711    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-workload-socket" pod "0626b947-6a73-4669-b426-b5b9ce1c30e3" (UID: "0626b947-6a73-4669-b426-b5b9ce1c30e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:07:54.213985 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.213747    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-workload-certs\") pod \"0626b947-6a73-4669-b426-b5b9ce1c30e3\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") "
Nov 03 22:07:54.213985 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.213793    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-envoy\") pod \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") "
Nov 03 22:07:54.213985 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.213827    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-workload-certs\") pod \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") "
Nov 03 22:07:54.213985 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.213861    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-podinfo\") pod \"0626b947-6a73-4669-b426-b5b9ce1c30e3\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") "
Nov 03 22:07:54.213985 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.213953    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-workload-socket\") pod \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\" (UID: \"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222\") "
Nov 03 22:07:54.214491 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.214015    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0626b947-6a73-4669-b426-b5b9ce1c30e3-kube-api-access-9sk44\" (UniqueName: \"kubernetes.io/projected/0626b947-6a73-4669-b426-b5b9ce1c30e3-kube-api-access-9sk44\") pod \"0626b947-6a73-4669-b426-b5b9ce1c30e3\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") "
Nov 03 22:07:54.214491 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.214046    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istiod-ca-cert" pod "5fb7d89c-af78-4e10-bfe9-ccbbb51e9222" (UID: "5fb7d89c-af78-4e10-bfe9-ccbbb51e9222"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:07:54.214491 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.214092    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-envoy\") pod \"0626b947-6a73-4669-b426-b5b9ce1c30e3\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") "
Nov 03 22:07:54.214491 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.214119    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-token\" (UniqueName: \"kubernetes.io/projected/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-token\") pod \"0626b947-6a73-4669-b426-b5b9ce1c30e3\" (UID: \"0626b947-6a73-4669-b426-b5b9ce1c30e3\") "
Nov 03 22:07:54.214491 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.214146    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-credential-socket" pod "0626b947-6a73-4669-b426-b5b9ce1c30e3" (UID: "0626b947-6a73-4669-b426-b5b9ce1c30e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:07:54.214491 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.214223    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-data" pod "0626b947-6a73-4669-b426-b5b9ce1c30e3" (UID: "0626b947-6a73-4669-b426-b5b9ce1c30e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:07:54.214491 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.214435    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-data" pod "5fb7d89c-af78-4e10-bfe9-ccbbb51e9222" (UID: "5fb7d89c-af78-4e10-bfe9-ccbbb51e9222"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:07:54.214491 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.214447    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-workload-socket" pod "5fb7d89c-af78-4e10-bfe9-ccbbb51e9222" (UID: "5fb7d89c-af78-4e10-bfe9-ccbbb51e9222"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:07:54.214491 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.214470    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0626b947-6a73-4669-b426-b5b9ce1c30e3-istiod-ca-cert" pod "0626b947-6a73-4669-b426-b5b9ce1c30e3" (UID: "0626b947-6a73-4669-b426-b5b9ce1c30e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:07:54.215005 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.214612    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-workload-certs" pod "5fb7d89c-af78-4e10-bfe9-ccbbb51e9222" (UID: "5fb7d89c-af78-4e10-bfe9-ccbbb51e9222"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:07:54.215005 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.214747    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:07:54.215005 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.214767    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:07:54.215005 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.214783    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:07:54.215005 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.214798    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:07:54.215005 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.214824    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:07:54.215005 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.214837    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0626b947-6a73-4669-b426-b5b9ce1c30e3-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:07:54.215005 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.214851    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:07:54.215005 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.214864    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:07:54.215005 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.214925    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-workload-certs" pod "0626b947-6a73-4669-b426-b5b9ce1c30e3" (UID: "0626b947-6a73-4669-b426-b5b9ce1c30e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:07:54.217296 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.217220    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0626b947-6a73-4669-b426-b5b9ce1c30e3-kube-api-access-9sk44" pod "0626b947-6a73-4669-b426-b5b9ce1c30e3" (UID: "0626b947-6a73-4669-b426-b5b9ce1c30e3"). InnerVolumeSpecName "kube-api-access-9sk44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:07:54.217441 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.217421    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-envoy" pod "5fb7d89c-af78-4e10-bfe9-ccbbb51e9222" (UID: "5fb7d89c-af78-4e10-bfe9-ccbbb51e9222"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:07:54.217606 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.217584    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-podinfo" pod "5fb7d89c-af78-4e10-bfe9-ccbbb51e9222" (UID: "5fb7d89c-af78-4e10-bfe9-ccbbb51e9222"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:07:54.217606 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.217601    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-token" pod "0626b947-6a73-4669-b426-b5b9ce1c30e3" (UID: "0626b947-6a73-4669-b426-b5b9ce1c30e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:07:54.217759 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.217626    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-kube-api-access-pkbd7" pod "5fb7d89c-af78-4e10-bfe9-ccbbb51e9222" (UID: "5fb7d89c-af78-4e10-bfe9-ccbbb51e9222"). InnerVolumeSpecName "kube-api-access-pkbd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:07:54.217759 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.217673    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-token" pod "5fb7d89c-af78-4e10-bfe9-ccbbb51e9222" (UID: "5fb7d89c-af78-4e10-bfe9-ccbbb51e9222"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:07:54.218755 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.218735    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-podinfo" pod "0626b947-6a73-4669-b426-b5b9ce1c30e3" (UID: "0626b947-6a73-4669-b426-b5b9ce1c30e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:07:54.219395 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.219379    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-envoy" pod "0626b947-6a73-4669-b426-b5b9ce1c30e3" (UID: "0626b947-6a73-4669-b426-b5b9ce1c30e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:07:54.316681 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.316644    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:07:54.316681 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.316678    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:07:54.316985 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.316691    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkbd7\" (UniqueName: \"kubernetes.io/projected/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-kube-api-access-pkbd7\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:07:54.316985 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.316704    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:07:54.316985 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.316717    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:07:54.316985 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.316728    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:07:54.316985 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.316738    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:07:54.316985 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.316750    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sk44\" (UniqueName: \"kubernetes.io/projected/0626b947-6a73-4669-b426-b5b9ce1c30e3-kube-api-access-9sk44\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:07:54.316985 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.316762    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:07:54.316985 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.316773    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0626b947-6a73-4669-b426-b5b9ce1c30e3-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:07:54.812384 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.812352    5053 generic.go:356] "Generic (PLEG): container finished" podID="0626b947-6a73-4669-b426-b5b9ce1c30e3" containerID="3e1f6328dcc2a3a03c4cdbae3a79687526a8864535f3ca532cc932acae33829c" exitCode=0
Nov 03 22:07:54.812874 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.812407    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"
Nov 03 22:07:54.812874 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.812434    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh" event={"ID":"0626b947-6a73-4669-b426-b5b9ce1c30e3","Type":"ContainerDied","Data":"3e1f6328dcc2a3a03c4cdbae3a79687526a8864535f3ca532cc932acae33829c"}
Nov 03 22:07:54.812874 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.812481    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh" event={"ID":"0626b947-6a73-4669-b426-b5b9ce1c30e3","Type":"ContainerDied","Data":"cb491dfea912d24796751ef8759652ec8f92a2b2078604ab8f87a421585a61ea"}
Nov 03 22:07:54.812874 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.812505    5053 scope.go:122] "RemoveContainer" containerID="3e1f6328dcc2a3a03c4cdbae3a79687526a8864535f3ca532cc932acae33829c"
Nov 03 22:07:54.814419 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.814389    5053 generic.go:356] "Generic (PLEG): container finished" podID="5fb7d89c-af78-4e10-bfe9-ccbbb51e9222" containerID="46a20bb03cfa578f8967ea82fcf7b0154b772d24ce057acac6e3ac9f4d8a60ea" exitCode=0
Nov 03 22:07:54.814549 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.814429    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx" event={"ID":"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222","Type":"ContainerDied","Data":"46a20bb03cfa578f8967ea82fcf7b0154b772d24ce057acac6e3ac9f4d8a60ea"}
Nov 03 22:07:54.814549 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.814453    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"
Nov 03 22:07:54.814549 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.814480    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx" event={"ID":"5fb7d89c-af78-4e10-bfe9-ccbbb51e9222","Type":"ContainerDied","Data":"fab0d0299576b6d9b4e0a31d26961e5d5505b32cce12a49c57ffd1ac9a76cae8"}
Nov 03 22:07:54.824094 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.824074    5053 scope.go:122] "RemoveContainer" containerID="3e1f6328dcc2a3a03c4cdbae3a79687526a8864535f3ca532cc932acae33829c"
Nov 03 22:07:54.824428 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:07:54.824397    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e1f6328dcc2a3a03c4cdbae3a79687526a8864535f3ca532cc932acae33829c\": container with ID starting with 3e1f6328dcc2a3a03c4cdbae3a79687526a8864535f3ca532cc932acae33829c not found: ID does not exist" containerID="3e1f6328dcc2a3a03c4cdbae3a79687526a8864535f3ca532cc932acae33829c"
Nov 03 22:07:54.824519 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.824442    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e1f6328dcc2a3a03c4cdbae3a79687526a8864535f3ca532cc932acae33829c"} err="failed to get container status \"3e1f6328dcc2a3a03c4cdbae3a79687526a8864535f3ca532cc932acae33829c\": rpc error: code = NotFound desc = could not find container \"3e1f6328dcc2a3a03c4cdbae3a79687526a8864535f3ca532cc932acae33829c\": container with ID starting with 3e1f6328dcc2a3a03c4cdbae3a79687526a8864535f3ca532cc932acae33829c not found: ID does not exist"
Nov 03 22:07:54.824519 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.824463    5053 scope.go:122] "RemoveContainer" containerID="46a20bb03cfa578f8967ea82fcf7b0154b772d24ce057acac6e3ac9f4d8a60ea"
Nov 03 22:07:54.835034 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.835016    5053 scope.go:122] "RemoveContainer" containerID="46a20bb03cfa578f8967ea82fcf7b0154b772d24ce057acac6e3ac9f4d8a60ea"
Nov 03 22:07:54.835347 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:07:54.835328    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46a20bb03cfa578f8967ea82fcf7b0154b772d24ce057acac6e3ac9f4d8a60ea\": container with ID starting with 46a20bb03cfa578f8967ea82fcf7b0154b772d24ce057acac6e3ac9f4d8a60ea not found: ID does not exist" containerID="46a20bb03cfa578f8967ea82fcf7b0154b772d24ce057acac6e3ac9f4d8a60ea"
Nov 03 22:07:54.835407 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.835355    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46a20bb03cfa578f8967ea82fcf7b0154b772d24ce057acac6e3ac9f4d8a60ea"} err="failed to get container status \"46a20bb03cfa578f8967ea82fcf7b0154b772d24ce057acac6e3ac9f4d8a60ea\": rpc error: code = NotFound desc = could not find container \"46a20bb03cfa578f8967ea82fcf7b0154b772d24ce057acac6e3ac9f4d8a60ea\": container with ID starting with 46a20bb03cfa578f8967ea82fcf7b0154b772d24ce057acac6e3ac9f4d8a60ea not found: ID does not exist"
Nov 03 22:07:54.978529 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:54.978494    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"]
Nov 03 22:07:55.029309 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:55.029185    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-pmkqh"]
Nov 03 22:07:55.090155 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:55.090118    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"]
Nov 03 22:07:55.094648 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:55.094619    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-bc2mx"]
Nov 03 22:07:55.669363 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:55.669331    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0626b947-6a73-4669-b426-b5b9ce1c30e3" path="/var/lib/kubelet/pods/0626b947-6a73-4669-b426-b5b9ce1c30e3/volumes"
Nov 03 22:07:55.669850 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:07:55.669835    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fb7d89c-af78-4e10-bfe9-ccbbb51e9222" path="/var/lib/kubelet/pods/5fb7d89c-af78-4e10-bfe9-ccbbb51e9222/volumes"
Nov 03 22:08:10.931225 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:08:10.931180    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:08:12.467537 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:12.467498    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"]
Nov 03 22:08:12.467964 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:12.467711    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2" podUID="978944c9-d91e-493f-8769-d6dacb9c679d" containerName="istio-proxy" containerID="cri-o://e6a38db1891311fd8358762e45bcd78549d4ccc5ef2ab9800ae8551e8433fc85" gracePeriod=30
Nov 03 22:08:12.517353 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:12.517313    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"]
Nov 03 22:08:12.517630 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:12.517590    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv" podUID="9a891ded-a561-4750-8f74-e85eed3f825f" containerName="istio-proxy" containerID="cri-o://c39e7735547ea83b6b69b945c601fb97812241b24fd78769e93a1f998a0608a9" gracePeriod=30
Nov 03 22:08:12.521197 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:12.521168    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"]
Nov 03 22:08:12.521439 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:12.521402    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb" podUID="cb5cb186-a9c9-491b-8798-77e9cb223d27" containerName="istio-proxy" containerID="cri-o://eba93d88f6109ce67d0d8c17d5609671f45dc9ad4679d274e90475c76a2e3c92" gracePeriod=30
Nov 03 22:08:17.346196 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.346156    5053 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:08:17.346651 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.346215    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv" podUID="9a891ded-a561-4750-8f74-e85eed3f825f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:08:17.833446 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.833410    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:08:17.891282 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.891252    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:08:17.899849 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.899820    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:08:17.938212 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.938178    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-istio-envoy\") pod \"978944c9-d91e-493f-8769-d6dacb9c679d\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") "
Nov 03 22:08:17.938424 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.938227    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-istio-data\") pod \"978944c9-d91e-493f-8769-d6dacb9c679d\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") "
Nov 03 22:08:17.938424 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.938303    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-credential-socket\") pod \"978944c9-d91e-493f-8769-d6dacb9c679d\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") "
Nov 03 22:08:17.938424 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.938342    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/978944c9-d91e-493f-8769-d6dacb9c679d-kube-api-access-gt6jf\" (UniqueName: \"kubernetes.io/projected/978944c9-d91e-493f-8769-d6dacb9c679d-kube-api-access-gt6jf\") pod \"978944c9-d91e-493f-8769-d6dacb9c679d\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") "
Nov 03 22:08:17.938424 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.938375    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/978944c9-d91e-493f-8769-d6dacb9c679d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/978944c9-d91e-493f-8769-d6dacb9c679d-istio-podinfo\") pod \"978944c9-d91e-493f-8769-d6dacb9c679d\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") "
Nov 03 22:08:17.938670 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.938453    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/978944c9-d91e-493f-8769-d6dacb9c679d-istio-token\" (UniqueName: \"kubernetes.io/projected/978944c9-d91e-493f-8769-d6dacb9c679d-istio-token\") pod \"978944c9-d91e-493f-8769-d6dacb9c679d\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") "
Nov 03 22:08:17.938670 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.938490    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-workload-socket\") pod \"978944c9-d91e-493f-8769-d6dacb9c679d\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") "
Nov 03 22:08:17.938670 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.938507    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-istio-data" pod "978944c9-d91e-493f-8769-d6dacb9c679d" (UID: "978944c9-d91e-493f-8769-d6dacb9c679d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:17.938670 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.938624    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-credential-socket" pod "978944c9-d91e-493f-8769-d6dacb9c679d" (UID: "978944c9-d91e-493f-8769-d6dacb9c679d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:17.938670 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.938640    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/978944c9-d91e-493f-8769-d6dacb9c679d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/978944c9-d91e-493f-8769-d6dacb9c679d-istiod-ca-cert\") pod \"978944c9-d91e-493f-8769-d6dacb9c679d\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") "
Nov 03 22:08:17.938954 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.938723    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-workload-socket" pod "978944c9-d91e-493f-8769-d6dacb9c679d" (UID: "978944c9-d91e-493f-8769-d6dacb9c679d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:17.938954 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.938789    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-workload-certs\") pod \"978944c9-d91e-493f-8769-d6dacb9c679d\" (UID: \"978944c9-d91e-493f-8769-d6dacb9c679d\") "
Nov 03 22:08:17.939074 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.939020    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978944c9-d91e-493f-8769-d6dacb9c679d-istiod-ca-cert" pod "978944c9-d91e-493f-8769-d6dacb9c679d" (UID: "978944c9-d91e-493f-8769-d6dacb9c679d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:08:17.939074 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.939027    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-workload-certs" pod "978944c9-d91e-493f-8769-d6dacb9c679d" (UID: "978944c9-d91e-493f-8769-d6dacb9c679d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:17.939729 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.939703    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:17.939849 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.939734    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:17.939849 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.939747    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:17.939849 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.939759    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:17.939849 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.939771    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/978944c9-d91e-493f-8769-d6dacb9c679d-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:17.941007 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.940968    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/978944c9-d91e-493f-8769-d6dacb9c679d-istio-podinfo" pod "978944c9-d91e-493f-8769-d6dacb9c679d" (UID: "978944c9-d91e-493f-8769-d6dacb9c679d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:08:17.941384 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.941358    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978944c9-d91e-493f-8769-d6dacb9c679d-istio-token" pod "978944c9-d91e-493f-8769-d6dacb9c679d" (UID: "978944c9-d91e-493f-8769-d6dacb9c679d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:08:17.941517 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.941496    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-istio-envoy" pod "978944c9-d91e-493f-8769-d6dacb9c679d" (UID: "978944c9-d91e-493f-8769-d6dacb9c679d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:17.941614 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:17.941580    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978944c9-d91e-493f-8769-d6dacb9c679d-kube-api-access-gt6jf" pod "978944c9-d91e-493f-8769-d6dacb9c679d" (UID: "978944c9-d91e-493f-8769-d6dacb9c679d"). InnerVolumeSpecName "kube-api-access-gt6jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:08:18.016760 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.016723    5053 generic.go:356] "Generic (PLEG): container finished" podID="cb5cb186-a9c9-491b-8798-77e9cb223d27" containerID="eba93d88f6109ce67d0d8c17d5609671f45dc9ad4679d274e90475c76a2e3c92" exitCode=0
Nov 03 22:08:18.016978 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.016784    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"
Nov 03 22:08:18.016978 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.016803    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb" event={"ID":"cb5cb186-a9c9-491b-8798-77e9cb223d27","Type":"ContainerDied","Data":"eba93d88f6109ce67d0d8c17d5609671f45dc9ad4679d274e90475c76a2e3c92"}
Nov 03 22:08:18.016978 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.016868    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb" event={"ID":"cb5cb186-a9c9-491b-8798-77e9cb223d27","Type":"ContainerDied","Data":"f86ba2f49835eee7ec41330a2d4542c95029c5240f713603d15aafe02a57755c"}
Nov 03 22:08:18.016978 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.016898    5053 scope.go:122] "RemoveContainer" containerID="eba93d88f6109ce67d0d8c17d5609671f45dc9ad4679d274e90475c76a2e3c92"
Nov 03 22:08:18.018907 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.018773    5053 generic.go:356] "Generic (PLEG): container finished" podID="9a891ded-a561-4750-8f74-e85eed3f825f" containerID="c39e7735547ea83b6b69b945c601fb97812241b24fd78769e93a1f998a0608a9" exitCode=0
Nov 03 22:08:18.019067 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.019032    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv" event={"ID":"9a891ded-a561-4750-8f74-e85eed3f825f","Type":"ContainerDied","Data":"c39e7735547ea83b6b69b945c601fb97812241b24fd78769e93a1f998a0608a9"}
Nov 03 22:08:18.019202 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.019094    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv" event={"ID":"9a891ded-a561-4750-8f74-e85eed3f825f","Type":"ContainerDied","Data":"66e19a45ff58f9d7e7777001da745834e67197e2eb87505d14d0f47487f6f995"}
Nov 03 22:08:18.019202 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.019170    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"
Nov 03 22:08:18.021092 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.021066    5053 generic.go:356] "Generic (PLEG): container finished" podID="978944c9-d91e-493f-8769-d6dacb9c679d" containerID="e6a38db1891311fd8358762e45bcd78549d4ccc5ef2ab9800ae8551e8433fc85" exitCode=0
Nov 03 22:08:18.021217 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.021120    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2" event={"ID":"978944c9-d91e-493f-8769-d6dacb9c679d","Type":"ContainerDied","Data":"e6a38db1891311fd8358762e45bcd78549d4ccc5ef2ab9800ae8551e8433fc85"}
Nov 03 22:08:18.021217 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.021160    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2" event={"ID":"978944c9-d91e-493f-8769-d6dacb9c679d","Type":"ContainerDied","Data":"0ec2cdc63d0a517ab2077b6be9e6f3a01352d3246dd2c4546d3ea9ce6d6b26c1"}
Nov 03 22:08:18.021426 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.021229    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"
Nov 03 22:08:18.031913 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.031892    5053 scope.go:122] "RemoveContainer" containerID="eba93d88f6109ce67d0d8c17d5609671f45dc9ad4679d274e90475c76a2e3c92"
Nov 03 22:08:18.032578 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:08:18.032276    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eba93d88f6109ce67d0d8c17d5609671f45dc9ad4679d274e90475c76a2e3c92\": container with ID starting with eba93d88f6109ce67d0d8c17d5609671f45dc9ad4679d274e90475c76a2e3c92 not found: ID does not exist" containerID="eba93d88f6109ce67d0d8c17d5609671f45dc9ad4679d274e90475c76a2e3c92"
Nov 03 22:08:18.032578 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.032316    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eba93d88f6109ce67d0d8c17d5609671f45dc9ad4679d274e90475c76a2e3c92"} err="failed to get container status \"eba93d88f6109ce67d0d8c17d5609671f45dc9ad4679d274e90475c76a2e3c92\": rpc error: code = NotFound desc = could not find container \"eba93d88f6109ce67d0d8c17d5609671f45dc9ad4679d274e90475c76a2e3c92\": container with ID starting with eba93d88f6109ce67d0d8c17d5609671f45dc9ad4679d274e90475c76a2e3c92 not found: ID does not exist"
Nov 03 22:08:18.032578 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.032338    5053 scope.go:122] "RemoveContainer" containerID="c39e7735547ea83b6b69b945c601fb97812241b24fd78769e93a1f998a0608a9"
Nov 03 22:08:18.040842 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.040817    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a891ded-a561-4750-8f74-e85eed3f825f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a891ded-a561-4750-8f74-e85eed3f825f-istiod-ca-cert\") pod \"9a891ded-a561-4750-8f74-e85eed3f825f\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") "
Nov 03 22:08:18.040987 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.040872    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a891ded-a561-4750-8f74-e85eed3f825f-istio-token\" (UniqueName: \"kubernetes.io/projected/9a891ded-a561-4750-8f74-e85eed3f825f-istio-token\") pod \"9a891ded-a561-4750-8f74-e85eed3f825f\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") "
Nov 03 22:08:18.040987 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.040914    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-istio-envoy\") pod \"9a891ded-a561-4750-8f74-e85eed3f825f\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") "
Nov 03 22:08:18.040987 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.040946    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a891ded-a561-4750-8f74-e85eed3f825f-kube-api-access-7v47t\" (UniqueName: \"kubernetes.io/projected/9a891ded-a561-4750-8f74-e85eed3f825f-kube-api-access-7v47t\") pod \"9a891ded-a561-4750-8f74-e85eed3f825f\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") "
Nov 03 22:08:18.040987 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.040979    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a891ded-a561-4750-8f74-e85eed3f825f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a891ded-a561-4750-8f74-e85eed3f825f-istio-podinfo\") pod \"9a891ded-a561-4750-8f74-e85eed3f825f\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") "
Nov 03 22:08:18.041716 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.041047    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-workload-socket\") pod \"cb5cb186-a9c9-491b-8798-77e9cb223d27\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") "
Nov 03 22:08:18.041716 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.041128    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-workload-certs\") pod \"cb5cb186-a9c9-491b-8798-77e9cb223d27\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") "
Nov 03 22:08:18.041716 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.041267    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a891ded-a561-4750-8f74-e85eed3f825f-istiod-ca-cert" pod "9a891ded-a561-4750-8f74-e85eed3f825f" (UID: "9a891ded-a561-4750-8f74-e85eed3f825f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:08:18.041716 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.041318    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-token\" (UniqueName: \"kubernetes.io/projected/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-token\") pod \"cb5cb186-a9c9-491b-8798-77e9cb223d27\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") "
Nov 03 22:08:18.041716 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.041401    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-envoy\") pod \"cb5cb186-a9c9-491b-8798-77e9cb223d27\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") "
Nov 03 22:08:18.041716 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.041428    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-data\") pod \"cb5cb186-a9c9-491b-8798-77e9cb223d27\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") "
Nov 03 22:08:18.041716 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.041472    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-workload-socket\") pod \"9a891ded-a561-4750-8f74-e85eed3f825f\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") "
Nov 03 22:08:18.041716 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.041510    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-podinfo\") pod \"cb5cb186-a9c9-491b-8798-77e9cb223d27\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") "
Nov 03 22:08:18.041716 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.041552    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb5cb186-a9c9-491b-8798-77e9cb223d27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb5cb186-a9c9-491b-8798-77e9cb223d27-istiod-ca-cert\") pod \"cb5cb186-a9c9-491b-8798-77e9cb223d27\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") "
Nov 03 22:08:18.041716 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.041595    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-istio-data\") pod \"9a891ded-a561-4750-8f74-e85eed3f825f\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") "
Nov 03 22:08:18.041716 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.041627    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-credential-socket\") pod \"cb5cb186-a9c9-491b-8798-77e9cb223d27\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") "
Nov 03 22:08:18.041716 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.041663    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-workload-certs\") pod \"9a891ded-a561-4750-8f74-e85eed3f825f\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") "
Nov 03 22:08:18.041716 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.041718    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb5cb186-a9c9-491b-8798-77e9cb223d27-kube-api-access-ds8ph\" (UniqueName: \"kubernetes.io/projected/cb5cb186-a9c9-491b-8798-77e9cb223d27-kube-api-access-ds8ph\") pod \"cb5cb186-a9c9-491b-8798-77e9cb223d27\" (UID: \"cb5cb186-a9c9-491b-8798-77e9cb223d27\") "
Nov 03 22:08:18.042374 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.041751    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-credential-socket\") pod \"9a891ded-a561-4750-8f74-e85eed3f825f\" (UID: \"9a891ded-a561-4750-8f74-e85eed3f825f\") "
Nov 03 22:08:18.042501 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.042472    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt6jf\" (UniqueName: \"kubernetes.io/projected/978944c9-d91e-493f-8769-d6dacb9c679d-kube-api-access-gt6jf\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:18.042617 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.042504    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/978944c9-d91e-493f-8769-d6dacb9c679d-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:18.042617 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.042519    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/978944c9-d91e-493f-8769-d6dacb9c679d-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:18.042617 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.042532    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a891ded-a561-4750-8f74-e85eed3f825f-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:18.042617 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.042547    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/978944c9-d91e-493f-8769-d6dacb9c679d-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:18.042834 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.042731    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-credential-socket" pod "9a891ded-a561-4750-8f74-e85eed3f825f" (UID: "9a891ded-a561-4750-8f74-e85eed3f825f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:18.042892 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.042848    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-workload-socket" pod "9a891ded-a561-4750-8f74-e85eed3f825f" (UID: "9a891ded-a561-4750-8f74-e85eed3f825f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:18.042982 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.042962    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-istio-data" pod "9a891ded-a561-4750-8f74-e85eed3f825f" (UID: "9a891ded-a561-4750-8f74-e85eed3f825f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:18.043044 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.042975    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-credential-socket" pod "cb5cb186-a9c9-491b-8798-77e9cb223d27" (UID: "cb5cb186-a9c9-491b-8798-77e9cb223d27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:18.043169 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.043152    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-workload-certs" pod "9a891ded-a561-4750-8f74-e85eed3f825f" (UID: "9a891ded-a561-4750-8f74-e85eed3f825f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:18.043549 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.043526    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb5cb186-a9c9-491b-8798-77e9cb223d27-istiod-ca-cert" pod "cb5cb186-a9c9-491b-8798-77e9cb223d27" (UID: "cb5cb186-a9c9-491b-8798-77e9cb223d27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:08:18.048662 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.048544    5053 scope.go:122] "RemoveContainer" containerID="c39e7735547ea83b6b69b945c601fb97812241b24fd78769e93a1f998a0608a9"
Nov 03 22:08:18.048787 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.048751    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-workload-socket" pod "cb5cb186-a9c9-491b-8798-77e9cb223d27" (UID: "cb5cb186-a9c9-491b-8798-77e9cb223d27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:18.048863 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.048774    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-data" pod "cb5cb186-a9c9-491b-8798-77e9cb223d27" (UID: "cb5cb186-a9c9-491b-8798-77e9cb223d27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:18.048863 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.048812    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-workload-certs" pod "cb5cb186-a9c9-491b-8798-77e9cb223d27" (UID: "cb5cb186-a9c9-491b-8798-77e9cb223d27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:18.049030 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:08:18.049009    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c39e7735547ea83b6b69b945c601fb97812241b24fd78769e93a1f998a0608a9\": container with ID starting with c39e7735547ea83b6b69b945c601fb97812241b24fd78769e93a1f998a0608a9 not found: ID does not exist" containerID="c39e7735547ea83b6b69b945c601fb97812241b24fd78769e93a1f998a0608a9"
Nov 03 22:08:18.049069 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.049040    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39e7735547ea83b6b69b945c601fb97812241b24fd78769e93a1f998a0608a9"} err="failed to get container status \"c39e7735547ea83b6b69b945c601fb97812241b24fd78769e93a1f998a0608a9\": rpc error: code = NotFound desc = could not find container \"c39e7735547ea83b6b69b945c601fb97812241b24fd78769e93a1f998a0608a9\": container with ID starting with c39e7735547ea83b6b69b945c601fb97812241b24fd78769e93a1f998a0608a9 not found: ID does not exist"
Nov 03 22:08:18.049069 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.049057    5053 scope.go:122] "RemoveContainer" containerID="e6a38db1891311fd8358762e45bcd78549d4ccc5ef2ab9800ae8551e8433fc85"
Nov 03 22:08:18.050855 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.050832    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-envoy" pod "cb5cb186-a9c9-491b-8798-77e9cb223d27" (UID: "cb5cb186-a9c9-491b-8798-77e9cb223d27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:18.051664 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.051638    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a891ded-a561-4750-8f74-e85eed3f825f-istio-token" pod "9a891ded-a561-4750-8f74-e85eed3f825f" (UID: "9a891ded-a561-4750-8f74-e85eed3f825f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:08:18.059608 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.059587    5053 scope.go:122] "RemoveContainer" containerID="e6a38db1891311fd8358762e45bcd78549d4ccc5ef2ab9800ae8551e8433fc85"
Nov 03 22:08:18.059956 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:08:18.059934    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6a38db1891311fd8358762e45bcd78549d4ccc5ef2ab9800ae8551e8433fc85\": container with ID starting with e6a38db1891311fd8358762e45bcd78549d4ccc5ef2ab9800ae8551e8433fc85 not found: ID does not exist" containerID="e6a38db1891311fd8358762e45bcd78549d4ccc5ef2ab9800ae8551e8433fc85"
Nov 03 22:08:18.060027 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.059966    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6a38db1891311fd8358762e45bcd78549d4ccc5ef2ab9800ae8551e8433fc85"} err="failed to get container status \"e6a38db1891311fd8358762e45bcd78549d4ccc5ef2ab9800ae8551e8433fc85\": rpc error: code = NotFound desc = could not find container \"e6a38db1891311fd8358762e45bcd78549d4ccc5ef2ab9800ae8551e8433fc85\": container with ID starting with e6a38db1891311fd8358762e45bcd78549d4ccc5ef2ab9800ae8551e8433fc85 not found: ID does not exist"
Nov 03 22:08:18.063761 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.063736    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a891ded-a561-4750-8f74-e85eed3f825f-istio-podinfo" pod "9a891ded-a561-4750-8f74-e85eed3f825f" (UID: "9a891ded-a561-4750-8f74-e85eed3f825f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:08:18.063761 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.063749    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-podinfo" pod "cb5cb186-a9c9-491b-8798-77e9cb223d27" (UID: "cb5cb186-a9c9-491b-8798-77e9cb223d27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:08:18.063903 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.063762    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-istio-envoy" pod "9a891ded-a561-4750-8f74-e85eed3f825f" (UID: "9a891ded-a561-4750-8f74-e85eed3f825f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:18.063903 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.063780    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a891ded-a561-4750-8f74-e85eed3f825f-kube-api-access-7v47t" pod "9a891ded-a561-4750-8f74-e85eed3f825f" (UID: "9a891ded-a561-4750-8f74-e85eed3f825f"). InnerVolumeSpecName "kube-api-access-7v47t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:08:18.063903 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.063802    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-token" pod "cb5cb186-a9c9-491b-8798-77e9cb223d27" (UID: "cb5cb186-a9c9-491b-8798-77e9cb223d27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:08:18.063903 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.063809    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb5cb186-a9c9-491b-8798-77e9cb223d27-kube-api-access-ds8ph" pod "cb5cb186-a9c9-491b-8798-77e9cb223d27" (UID: "cb5cb186-a9c9-491b-8798-77e9cb223d27"). InnerVolumeSpecName "kube-api-access-ds8ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:08:18.144507 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.144397    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:18.144507 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.144436    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb5cb186-a9c9-491b-8798-77e9cb223d27-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:18.144507 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.144450    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:18.144507 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.144465    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:18.144507 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.144479    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:18.144507 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.144493    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds8ph\" (UniqueName: \"kubernetes.io/projected/cb5cb186-a9c9-491b-8798-77e9cb223d27-kube-api-access-ds8ph\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:18.144507 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.144508    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:18.145107 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.144522    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a891ded-a561-4750-8f74-e85eed3f825f-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:18.145107 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.144537    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:18.145107 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.144550    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v47t\" (UniqueName: \"kubernetes.io/projected/9a891ded-a561-4750-8f74-e85eed3f825f-kube-api-access-7v47t\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:18.145107 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.144563    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a891ded-a561-4750-8f74-e85eed3f825f-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:18.145107 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.144576    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:18.145107 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.144592    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:18.145107 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.144605    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:18.145107 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.144618    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:18.145107 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.144632    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb5cb186-a9c9-491b-8798-77e9cb223d27-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:18.145107 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.144649    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a891ded-a561-4750-8f74-e85eed3f825f-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:18.453766 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.453734    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"]
Nov 03 22:08:18.458528 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.458496    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-8v8k2"]
Nov 03 22:08:18.567016 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.566979    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"]
Nov 03 22:08:18.573982 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.573945    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-hctnb"]
Nov 03 22:08:18.680625 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.680589    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"]
Nov 03 22:08:18.685616 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:18.685581    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-q65kv"]
Nov 03 22:08:19.681752 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:19.681719    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="978944c9-d91e-493f-8769-d6dacb9c679d" path="/var/lib/kubelet/pods/978944c9-d91e-493f-8769-d6dacb9c679d/volumes"
Nov 03 22:08:19.682434 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:19.682417    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a891ded-a561-4750-8f74-e85eed3f825f" path="/var/lib/kubelet/pods/9a891ded-a561-4750-8f74-e85eed3f825f/volumes"
Nov 03 22:08:19.682864 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:19.682851    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb5cb186-a9c9-491b-8798-77e9cb223d27" path="/var/lib/kubelet/pods/cb5cb186-a9c9-491b-8798-77e9cb223d27/volumes"
Nov 03 22:08:36.551747 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.551704    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"]
Nov 03 22:08:36.552410 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.552378    5053 cpu_manager.go:414] "containerMap: removing container" podUID="978944c9-d91e-493f-8769-d6dacb9c679d" containerName="istio-proxy"
Nov 03 22:08:36.552410 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.552394    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="978944c9-d91e-493f-8769-d6dacb9c679d" containerName="istio-proxy"
Nov 03 22:08:36.552410 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.552411    5053 cpu_manager.go:414] "containerMap: removing container" podUID="cb5cb186-a9c9-491b-8798-77e9cb223d27" containerName="istio-proxy"
Nov 03 22:08:36.552575 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.552419    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb5cb186-a9c9-491b-8798-77e9cb223d27" containerName="istio-proxy"
Nov 03 22:08:36.552575 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.552440    5053 cpu_manager.go:414] "containerMap: removing container" podUID="9a891ded-a561-4750-8f74-e85eed3f825f" containerName="istio-proxy"
Nov 03 22:08:36.552575 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.552446    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a891ded-a561-4750-8f74-e85eed3f825f" containerName="istio-proxy"
Nov 03 22:08:36.552575 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.552480    5053 cpu_manager.go:414] "containerMap: removing container" podUID="0626b947-6a73-4669-b426-b5b9ce1c30e3" containerName="istio-proxy"
Nov 03 22:08:36.552575 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.552486    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0626b947-6a73-4669-b426-b5b9ce1c30e3" containerName="istio-proxy"
Nov 03 22:08:36.552575 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.552492    5053 cpu_manager.go:414] "containerMap: removing container" podUID="5fb7d89c-af78-4e10-bfe9-ccbbb51e9222" containerName="istio-proxy"
Nov 03 22:08:36.552575 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.552498    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fb7d89c-af78-4e10-bfe9-ccbbb51e9222" containerName="istio-proxy"
Nov 03 22:08:36.552920 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.552593    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb5cb186-a9c9-491b-8798-77e9cb223d27" containerName="istio-proxy"
Nov 03 22:08:36.552920 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.552606    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a891ded-a561-4750-8f74-e85eed3f825f" containerName="istio-proxy"
Nov 03 22:08:36.552920 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.552622    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="0626b947-6a73-4669-b426-b5b9ce1c30e3" containerName="istio-proxy"
Nov 03 22:08:36.552920 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.552631    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fb7d89c-af78-4e10-bfe9-ccbbb51e9222" containerName="istio-proxy"
Nov 03 22:08:36.552920 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.552639    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="978944c9-d91e-493f-8769-d6dacb9c679d" containerName="istio-proxy"
Nov 03 22:08:36.557882 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.557858    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:08:36.619862 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.619820    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"]
Nov 03 22:08:36.663412 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.663373    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-27a3c3\"/\"openshift-service-ca.crt\""
Nov 03 22:08:36.663412 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.663377    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-27a3c3\"/\"kube-root-ca.crt\""
Nov 03 22:08:36.663693 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.663429    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-27a3c3\"/\"default-dockercfg-nhnjm\""
Nov 03 22:08:36.663693 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.663469    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-27a3c3\"/\"explicit-model-27a3c3-kserve-self-signed-certs\""
Nov 03 22:08:36.687246 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.687186    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-dshm\") pod \"explicit-model-27a3c3-kserve-87fd9d869-p78nq\" (UID: \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\") " pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:08:36.687488 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.687358    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-home\") pod \"explicit-model-27a3c3-kserve-87fd9d869-p78nq\" (UID: \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\") " pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:08:36.687488 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.687385    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-tmp-dir\") pod \"explicit-model-27a3c3-kserve-87fd9d869-p78nq\" (UID: \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\") " pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:08:36.687488 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.687431    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-model-cache\") pod \"explicit-model-27a3c3-kserve-87fd9d869-p78nq\" (UID: \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\") " pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:08:36.687630 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.687598    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-487nm\" (UniqueName: \"kubernetes.io/projected/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-kube-api-access-487nm\") pod \"explicit-model-27a3c3-kserve-87fd9d869-p78nq\" (UID: \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\") " pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:08:36.687723 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.687702    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-tls-certs\") pod \"explicit-model-27a3c3-kserve-87fd9d869-p78nq\" (UID: \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\") " pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:08:36.789174 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.789134    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-home\") pod \"explicit-model-27a3c3-kserve-87fd9d869-p78nq\" (UID: \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\") " pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:08:36.789174 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.789176    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-tmp-dir\") pod \"explicit-model-27a3c3-kserve-87fd9d869-p78nq\" (UID: \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\") " pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:08:36.789508 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.789223    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-model-cache\") pod \"explicit-model-27a3c3-kserve-87fd9d869-p78nq\" (UID: \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\") " pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:08:36.789508 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.789366    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-487nm\" (UniqueName: \"kubernetes.io/projected/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-kube-api-access-487nm\") pod \"explicit-model-27a3c3-kserve-87fd9d869-p78nq\" (UID: \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\") " pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:08:36.789508 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.789412    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-tls-certs\") pod \"explicit-model-27a3c3-kserve-87fd9d869-p78nq\" (UID: \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\") " pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:08:36.789508 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.789479    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-dshm\") pod \"explicit-model-27a3c3-kserve-87fd9d869-p78nq\" (UID: \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\") " pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:08:36.789740 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.789705    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-home\") pod \"explicit-model-27a3c3-kserve-87fd9d869-p78nq\" (UID: \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\") " pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:08:36.789786 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.789771    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-tmp-dir\") pod \"explicit-model-27a3c3-kserve-87fd9d869-p78nq\" (UID: \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\") " pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:08:36.789950 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.789924    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-model-cache\") pod \"explicit-model-27a3c3-kserve-87fd9d869-p78nq\" (UID: \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\") " pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:08:36.791945 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.791925    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-dshm\") pod \"explicit-model-27a3c3-kserve-87fd9d869-p78nq\" (UID: \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\") " pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:08:36.792424 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.792402    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-tls-certs\") pod \"explicit-model-27a3c3-kserve-87fd9d869-p78nq\" (UID: \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\") " pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:08:36.847040 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.846953    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-487nm\" (UniqueName: \"kubernetes.io/projected/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-kube-api-access-487nm\") pod \"explicit-model-27a3c3-kserve-87fd9d869-p78nq\" (UID: \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\") " pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:08:36.871336 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:36.871295    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:08:37.253516 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:08:37.253482    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8d2ae15_538a_4eef_9dfd_1161ec25bce9.slice/crio-62f0b9a33836c6e46829846fa53596483d09e3ab9a1d2bb9663105cdce7563ea WatchSource:0}: Error finding container 62f0b9a33836c6e46829846fa53596483d09e3ab9a1d2bb9663105cdce7563ea: Status 404 returned error can't find the container with id 62f0b9a33836c6e46829846fa53596483d09e3ab9a1d2bb9663105cdce7563ea
Nov 03 22:08:37.253978 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:37.253954    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"]
Nov 03 22:08:38.196606 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:38.196567    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq" event={"ID":"b8d2ae15-538a-4eef-9dfd-1161ec25bce9","Type":"ContainerStarted","Data":"14e4571bf6d5c26ccbc33c727d69dae947ecbe2e38376f2663e44d06c65c9642"}
Nov 03 22:08:38.197034 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:38.196637    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq" event={"ID":"b8d2ae15-538a-4eef-9dfd-1161ec25bce9","Type":"ContainerStarted","Data":"62f0b9a33836c6e46829846fa53596483d09e3ab9a1d2bb9663105cdce7563ea"}
Nov 03 22:08:38.312066 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:38.312003    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq" podStartSLOduration=2.3119832430000002 podStartE2EDuration="2.311983243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:08:38.310611244 +0000 UTC m=+3232.804533120" watchObservedRunningTime="2026-11-03 22:08:38.311983243 +0000 UTC m=+3232.805905126"
Nov 03 22:08:38.871983 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:38.871929    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:08:40.930671 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:08:40.930624    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:08:46.872394 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:46.872347    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:08:46.885995 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:46.885964    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:08:47.281962 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:47.281924    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:08:47.957292 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:08:47.957224    5053 scope.go:122] "RemoveContainer" containerID="f3bdb9151729a99e3ebf8e5d4ffc1a8b639822dcdfe09e4fd9771fe30e6eecd2"
Nov 03 22:09:10.931076 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:09:10.931033    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:09:38.671188 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:38.671148    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"]
Nov 03 22:09:40.930467 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:09:40.930421    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:09:46.230414 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:46.230378    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 22:09:46.237451 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:46.237423    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 22:09:46.238608 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:46.238589    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 22:09:46.247167 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:46.247142    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 22:09:48.013808 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:48.013764    5053 scope.go:122] "RemoveContainer" containerID="3044d098994c8a9d2da7d365972dd5f3c8613ddf44c6588f4707ce4fee3cddc2"
Nov 03 22:09:53.692752 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:53.692678    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq" podUID="b8d2ae15-538a-4eef-9dfd-1161ec25bce9" containerName="main" containerID="cri-o://14e4571bf6d5c26ccbc33c727d69dae947ecbe2e38376f2663e44d06c65c9642" gracePeriod=45
Nov 03 22:09:54.000105 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.000078    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:09:54.080493 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.080456    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-tls-certs\") pod \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\" (UID: \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\") "
Nov 03 22:09:54.080684 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.080511    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-kube-api-access-487nm\" (UniqueName: \"kubernetes.io/projected/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-kube-api-access-487nm\") pod \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\" (UID: \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\") "
Nov 03 22:09:54.080684 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.080588    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-home\" (UniqueName: \"kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-home\") pod \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\" (UID: \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\") "
Nov 03 22:09:54.080684 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.080617    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-model-cache\") pod \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\" (UID: \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\") "
Nov 03 22:09:54.080855 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.080708    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-dshm\") pod \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\" (UID: \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\") "
Nov 03 22:09:54.080855 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.080845    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-tmp-dir\") pod \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\" (UID: \"b8d2ae15-538a-4eef-9dfd-1161ec25bce9\") "
Nov 03 22:09:54.080975 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.080882    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-home" pod "b8d2ae15-538a-4eef-9dfd-1161ec25bce9" (UID: "b8d2ae15-538a-4eef-9dfd-1161ec25bce9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:09:54.081040 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.080986    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-model-cache" pod "b8d2ae15-538a-4eef-9dfd-1161ec25bce9" (UID: "b8d2ae15-538a-4eef-9dfd-1161ec25bce9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:09:54.081156 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.081118    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-tmp-dir" pod "b8d2ae15-538a-4eef-9dfd-1161ec25bce9" (UID: "b8d2ae15-538a-4eef-9dfd-1161ec25bce9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:09:54.081495 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.081391    5053 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-tmp-dir\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:09:54.081495 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.081406    5053 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-home\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:09:54.081495 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.081415    5053 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-model-cache\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:09:54.083058 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.083028    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-tls-certs" pod "b8d2ae15-538a-4eef-9dfd-1161ec25bce9" (UID: "b8d2ae15-538a-4eef-9dfd-1161ec25bce9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:09:54.083058 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.083046    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-dshm" pod "b8d2ae15-538a-4eef-9dfd-1161ec25bce9" (UID: "b8d2ae15-538a-4eef-9dfd-1161ec25bce9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:09:54.083191 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.083083    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-kube-api-access-487nm" pod "b8d2ae15-538a-4eef-9dfd-1161ec25bce9" (UID: "b8d2ae15-538a-4eef-9dfd-1161ec25bce9"). InnerVolumeSpecName "kube-api-access-487nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:09:54.182598 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.182559    5053 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-tls-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:09:54.182598 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.182598    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-487nm\" (UniqueName: \"kubernetes.io/projected/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-kube-api-access-487nm\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:09:54.182901 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.182613    5053 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8d2ae15-538a-4eef-9dfd-1161ec25bce9-dshm\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:09:54.781117 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.781078    5053 generic.go:356] "Generic (PLEG): container finished" podID="b8d2ae15-538a-4eef-9dfd-1161ec25bce9" containerID="14e4571bf6d5c26ccbc33c727d69dae947ecbe2e38376f2663e44d06c65c9642" exitCode=0
Nov 03 22:09:54.781117 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.781124    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"
Nov 03 22:09:54.781726 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.781174    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq" event={"ID":"b8d2ae15-538a-4eef-9dfd-1161ec25bce9","Type":"ContainerDied","Data":"14e4571bf6d5c26ccbc33c727d69dae947ecbe2e38376f2663e44d06c65c9642"}
Nov 03 22:09:54.781726 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.781221    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq" event={"ID":"b8d2ae15-538a-4eef-9dfd-1161ec25bce9","Type":"ContainerDied","Data":"62f0b9a33836c6e46829846fa53596483d09e3ab9a1d2bb9663105cdce7563ea"}
Nov 03 22:09:54.781726 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.781259    5053 scope.go:122] "RemoveContainer" containerID="14e4571bf6d5c26ccbc33c727d69dae947ecbe2e38376f2663e44d06c65c9642"
Nov 03 22:09:54.792569 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.792548    5053 scope.go:122] "RemoveContainer" containerID="14e4571bf6d5c26ccbc33c727d69dae947ecbe2e38376f2663e44d06c65c9642"
Nov 03 22:09:54.792861 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:09:54.792839    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14e4571bf6d5c26ccbc33c727d69dae947ecbe2e38376f2663e44d06c65c9642\": container with ID starting with 14e4571bf6d5c26ccbc33c727d69dae947ecbe2e38376f2663e44d06c65c9642 not found: ID does not exist" containerID="14e4571bf6d5c26ccbc33c727d69dae947ecbe2e38376f2663e44d06c65c9642"
Nov 03 22:09:54.792918 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.792869    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14e4571bf6d5c26ccbc33c727d69dae947ecbe2e38376f2663e44d06c65c9642"} err="failed to get container status \"14e4571bf6d5c26ccbc33c727d69dae947ecbe2e38376f2663e44d06c65c9642\": rpc error: code = NotFound desc = could not find container \"14e4571bf6d5c26ccbc33c727d69dae947ecbe2e38376f2663e44d06c65c9642\": container with ID starting with 14e4571bf6d5c26ccbc33c727d69dae947ecbe2e38376f2663e44d06c65c9642 not found: ID does not exist"
Nov 03 22:09:54.945762 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.945725    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"]
Nov 03 22:09:54.994860 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:54.994819    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-27a3c3/explicit-model-27a3c3-kserve-87fd9d869-p78nq"]
Nov 03 22:09:55.668209 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:09:55.668170    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8d2ae15-538a-4eef-9dfd-1161ec25bce9" path="/var/lib/kubelet/pods/b8d2ae15-538a-4eef-9dfd-1161ec25bce9/volumes"
Nov 03 22:10:02.407737 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.407689    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"]
Nov 03 22:10:02.409193 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.409161    5053 cpu_manager.go:414] "containerMap: removing container" podUID="b8d2ae15-538a-4eef-9dfd-1161ec25bce9" containerName="main"
Nov 03 22:10:02.409193 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.409192    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d2ae15-538a-4eef-9dfd-1161ec25bce9" containerName="main"
Nov 03 22:10:02.409603 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.409585    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d2ae15-538a-4eef-9dfd-1161ec25bce9" containerName="main"
Nov 03 22:10:02.416066 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.416032    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:10:02.467007 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.466971    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"]
Nov 03 22:10:02.501644 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.501605    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-dshm\") pod \"auth-test-model-75f577-kserve-5cff888bbf-sqpnr\" (UID: \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\") " pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:10:02.501644 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.501644    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lxxq\" (UniqueName: \"kubernetes.io/projected/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-kube-api-access-9lxxq\") pod \"auth-test-model-75f577-kserve-5cff888bbf-sqpnr\" (UID: \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\") " pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:10:02.501931 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.501669    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-home\") pod \"auth-test-model-75f577-kserve-5cff888bbf-sqpnr\" (UID: \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\") " pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:10:02.501931 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.501841    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-model-cache\") pod \"auth-test-model-75f577-kserve-5cff888bbf-sqpnr\" (UID: \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\") " pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:10:02.501931 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.501895    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-tmp-dir\") pod \"auth-test-model-75f577-kserve-5cff888bbf-sqpnr\" (UID: \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\") " pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:10:02.502127 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.501956    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-tls-certs\") pod \"auth-test-model-75f577-kserve-5cff888bbf-sqpnr\" (UID: \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\") " pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:10:02.512015 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.511984    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-75f577\"/\"kube-root-ca.crt\""
Nov 03 22:10:02.512015 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.511984    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-75f577\"/\"openshift-service-ca.crt\""
Nov 03 22:10:02.517795 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.517769    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-75f577\"/\"default-dockercfg-x9k8p\""
Nov 03 22:10:02.517895 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.517769    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-75f577\"/\"auth-test-model-75f577-kserve-self-signed-certs\""
Nov 03 22:10:02.602898 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.602861    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-dshm\") pod \"auth-test-model-75f577-kserve-5cff888bbf-sqpnr\" (UID: \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\") " pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:10:02.602898 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.602904    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lxxq\" (UniqueName: \"kubernetes.io/projected/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-kube-api-access-9lxxq\") pod \"auth-test-model-75f577-kserve-5cff888bbf-sqpnr\" (UID: \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\") " pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:10:02.603206 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.602929    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-home\") pod \"auth-test-model-75f577-kserve-5cff888bbf-sqpnr\" (UID: \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\") " pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:10:02.603206 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.602964    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-model-cache\") pod \"auth-test-model-75f577-kserve-5cff888bbf-sqpnr\" (UID: \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\") " pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:10:02.603206 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.602993    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-tmp-dir\") pod \"auth-test-model-75f577-kserve-5cff888bbf-sqpnr\" (UID: \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\") " pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:10:02.603206 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.603022    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-tls-certs\") pod \"auth-test-model-75f577-kserve-5cff888bbf-sqpnr\" (UID: \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\") " pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:10:02.603485 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.603449    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-model-cache\") pod \"auth-test-model-75f577-kserve-5cff888bbf-sqpnr\" (UID: \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\") " pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:10:02.603535 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.603506    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-home\") pod \"auth-test-model-75f577-kserve-5cff888bbf-sqpnr\" (UID: \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\") " pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:10:02.603588 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.603567    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-tmp-dir\") pod \"auth-test-model-75f577-kserve-5cff888bbf-sqpnr\" (UID: \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\") " pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:10:02.605620 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.605590    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-dshm\") pod \"auth-test-model-75f577-kserve-5cff888bbf-sqpnr\" (UID: \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\") " pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:10:02.605864 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.605844    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-tls-certs\") pod \"auth-test-model-75f577-kserve-5cff888bbf-sqpnr\" (UID: \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\") " pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:10:02.664613 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.664540    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lxxq\" (UniqueName: \"kubernetes.io/projected/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-kube-api-access-9lxxq\") pod \"auth-test-model-75f577-kserve-5cff888bbf-sqpnr\" (UID: \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\") " pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:10:02.729157 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:02.729114    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:10:03.113029 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:03.112994    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"]
Nov 03 22:10:03.115022 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:10:03.114990    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f6f96cd_70e3_445b_8ccf_9d305d5f6ee9.slice/crio-3854ef6422dbafbcdf2efbcbee60897c6aa708b31bb111f7d79f6a066e8f6e1d WatchSource:0}: Error finding container 3854ef6422dbafbcdf2efbcbee60897c6aa708b31bb111f7d79f6a066e8f6e1d: Status 404 returned error can't find the container with id 3854ef6422dbafbcdf2efbcbee60897c6aa708b31bb111f7d79f6a066e8f6e1d
Nov 03 22:10:03.873976 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:03.873941    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr" event={"ID":"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9","Type":"ContainerStarted","Data":"239708540b812c4ef9cc3a211b873d04327cdc43a273a344002f92710003f619"}
Nov 03 22:10:03.873976 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:03.873981    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr" event={"ID":"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9","Type":"ContainerStarted","Data":"3854ef6422dbafbcdf2efbcbee60897c6aa708b31bb111f7d79f6a066e8f6e1d"}
Nov 03 22:10:03.993046 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:03.992989    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr" podStartSLOduration=1.992972636 podStartE2EDuration="1.992972636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:10:03.992400392 +0000 UTC m=+3318.486322267" watchObservedRunningTime="2026-11-03 22:10:03.992972636 +0000 UTC m=+3318.486894511"
Nov 03 22:10:04.729834 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:04.729797    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:10:05.653204 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:05.653162    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7676-e1f049"]
Nov 03 22:10:05.658548 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:05.658520    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7676-e1f049"
Nov 03 22:10:05.710318 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:05.710264    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7676-e1f049"]
Nov 03 22:10:05.755478 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:05.755440    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8dgqp\""
Nov 03 22:10:05.852268 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:05.852204    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plf72\" (UniqueName: \"kubernetes.io/projected/c35172ca-45a9-42da-a7dc-645dad791743-kube-api-access-plf72\") pod \"test-curl-7676-e1f049\" (UID: \"c35172ca-45a9-42da-a7dc-645dad791743\") " pod="opendatahub/test-curl-7676-e1f049"
Nov 03 22:10:05.953167 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:05.953127    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plf72\" (UniqueName: \"kubernetes.io/projected/c35172ca-45a9-42da-a7dc-645dad791743-kube-api-access-plf72\") pod \"test-curl-7676-e1f049\" (UID: \"c35172ca-45a9-42da-a7dc-645dad791743\") " pod="opendatahub/test-curl-7676-e1f049"
Nov 03 22:10:06.009518 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:06.009481    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plf72\" (UniqueName: \"kubernetes.io/projected/c35172ca-45a9-42da-a7dc-645dad791743-kube-api-access-plf72\") pod \"test-curl-7676-e1f049\" (UID: \"c35172ca-45a9-42da-a7dc-645dad791743\") " pod="opendatahub/test-curl-7676-e1f049"
Nov 03 22:10:06.274520 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:06.274425    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7676-e1f049"
Nov 03 22:10:06.656530 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:06.656503    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7676-e1f049"]
Nov 03 22:10:06.656964 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:10:06.656622    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc35172ca_45a9_42da_a7dc_645dad791743.slice/crio-2ff7bcdc5a8ab54ee38b0250166cb59bbecf6e48e52426226af6bde949927609 WatchSource:0}: Error finding container 2ff7bcdc5a8ab54ee38b0250166cb59bbecf6e48e52426226af6bde949927609: Status 404 returned error can't find the container with id 2ff7bcdc5a8ab54ee38b0250166cb59bbecf6e48e52426226af6bde949927609
Nov 03 22:10:06.896599 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:06.896561    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7676-e1f049" event={"ID":"c35172ca-45a9-42da-a7dc-645dad791743","Type":"ContainerStarted","Data":"093a70a40e1b8603953d35cc2563732f1967e1b8734fb6b6404df68a0a658d58"}
Nov 03 22:10:06.896824 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:06.896617    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7676-e1f049" event={"ID":"c35172ca-45a9-42da-a7dc-645dad791743","Type":"ContainerStarted","Data":"2ff7bcdc5a8ab54ee38b0250166cb59bbecf6e48e52426226af6bde949927609"}
Nov 03 22:10:07.009825 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:07.009760    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7676-e1f049" podStartSLOduration=2.009746046 podStartE2EDuration="2.009746046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:10:07.007336197 +0000 UTC m=+3321.501258073" watchObservedRunningTime="2026-11-03 22:10:07.009746046 +0000 UTC m=+3321.503667922"
Nov 03 22:10:08.499523 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:08.499388    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7676-e1f049"]
Nov 03 22:10:08.503665 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:08.503635    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7676-e1f049"]
Nov 03 22:10:08.909759 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:08.909667    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7676-e1f049" podUID="c35172ca-45a9-42da-a7dc-645dad791743" containerName="test-curl-7676-e1f049" containerID="cri-o://093a70a40e1b8603953d35cc2563732f1967e1b8734fb6b6404df68a0a658d58" gracePeriod=2
Nov 03 22:10:08.960675 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:10:08.960629    5053 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7676-e1f049\" is forbidden: User \"system:node:ip-10-0-128-43.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-43.us-west-1.compute.internal' and this object" podUID="c35172ca-45a9-42da-a7dc-645dad791743" pod="opendatahub/test-curl-7676-e1f049"
Nov 03 22:10:10.932642 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:10:10.932614    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:10:11.143365 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:11.143332    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7676-e1f049"
Nov 03 22:10:11.227264 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:11.227191    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c35172ca-45a9-42da-a7dc-645dad791743-kube-api-access-plf72\" (UniqueName: \"kubernetes.io/projected/c35172ca-45a9-42da-a7dc-645dad791743-kube-api-access-plf72\") pod \"c35172ca-45a9-42da-a7dc-645dad791743\" (UID: \"c35172ca-45a9-42da-a7dc-645dad791743\") "
Nov 03 22:10:11.229591 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:11.229563    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35172ca-45a9-42da-a7dc-645dad791743-kube-api-access-plf72" pod "c35172ca-45a9-42da-a7dc-645dad791743" (UID: "c35172ca-45a9-42da-a7dc-645dad791743"). InnerVolumeSpecName "kube-api-access-plf72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:10:11.328712 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:11.328674    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plf72\" (UniqueName: \"kubernetes.io/projected/c35172ca-45a9-42da-a7dc-645dad791743-kube-api-access-plf72\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:11.668103 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:11.668022    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c35172ca-45a9-42da-a7dc-645dad791743" path="/var/lib/kubelet/pods/c35172ca-45a9-42da-a7dc-645dad791743/volumes"
Nov 03 22:10:11.935132 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:11.935035    5053 generic.go:356] "Generic (PLEG): container finished" podID="c35172ca-45a9-42da-a7dc-645dad791743" containerID="093a70a40e1b8603953d35cc2563732f1967e1b8734fb6b6404df68a0a658d58" exitCode=137
Nov 03 22:10:11.935132 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:11.935094    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7676-e1f049"
Nov 03 22:10:11.935132 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:11.935112    5053 scope.go:122] "RemoveContainer" containerID="093a70a40e1b8603953d35cc2563732f1967e1b8734fb6b6404df68a0a658d58"
Nov 03 22:10:11.946947 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:11.946912    5053 scope.go:122] "RemoveContainer" containerID="093a70a40e1b8603953d35cc2563732f1967e1b8734fb6b6404df68a0a658d58"
Nov 03 22:10:11.947336 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:10:11.947312    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"093a70a40e1b8603953d35cc2563732f1967e1b8734fb6b6404df68a0a658d58\": container with ID starting with 093a70a40e1b8603953d35cc2563732f1967e1b8734fb6b6404df68a0a658d58 not found: ID does not exist" containerID="093a70a40e1b8603953d35cc2563732f1967e1b8734fb6b6404df68a0a658d58"
Nov 03 22:10:11.947444 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:11.947350    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"093a70a40e1b8603953d35cc2563732f1967e1b8734fb6b6404df68a0a658d58"} err="failed to get container status \"093a70a40e1b8603953d35cc2563732f1967e1b8734fb6b6404df68a0a658d58\": rpc error: code = NotFound desc = could not find container \"093a70a40e1b8603953d35cc2563732f1967e1b8734fb6b6404df68a0a658d58\": container with ID starting with 093a70a40e1b8603953d35cc2563732f1967e1b8734fb6b6404df68a0a658d58 not found: ID does not exist"
Nov 03 22:10:12.047464 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:12.047425    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b97c67-pl9qn"]
Nov 03 22:10:12.096211 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:12.096172    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b97c67-pl9qn"]
Nov 03 22:10:12.729628 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:12.729589    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:10:12.743891 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:12.743858    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:10:12.960818 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:12.960786    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:10:13.669944 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:13.669905    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b07179fa-cced-46f1-8952-8fe700769c7a" path="/var/lib/kubelet/pods/b07179fa-cced-46f1-8952-8fe700769c7a/volumes"
Nov 03 22:10:16.590918 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:16.590882    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"]
Nov 03 22:10:16.591419 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:16.591162    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn" podUID="ce0ed29f-9f1e-4ac2-80d5-b806fa23a225" containerName="maas-api" containerID="cri-o://ede53729f16145eeebcb079a9f48bc0d716a5a2d9170a43615c543c8de74a9fc" gracePeriod=30
Nov 03 22:10:16.908081 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:16.908037    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"
Nov 03 22:10:16.977747 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:16.977710    5053 generic.go:356] "Generic (PLEG): container finished" podID="ce0ed29f-9f1e-4ac2-80d5-b806fa23a225" containerID="ede53729f16145eeebcb079a9f48bc0d716a5a2d9170a43615c543c8de74a9fc" exitCode=0
Nov 03 22:10:16.978049 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:16.978000    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn" event={"ID":"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225","Type":"ContainerDied","Data":"ede53729f16145eeebcb079a9f48bc0d716a5a2d9170a43615c543c8de74a9fc"}
Nov 03 22:10:16.978155 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:16.978062    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn" event={"ID":"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225","Type":"ContainerDied","Data":"4294b0d3503a845c7972706f7e5ed5f1b0bb947bcb277eeca6ea749519f4f25e"}
Nov 03 22:10:16.978155 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:16.978095    5053 scope.go:122] "RemoveContainer" containerID="ede53729f16145eeebcb079a9f48bc0d716a5a2d9170a43615c543c8de74a9fc"
Nov 03 22:10:16.978902 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:16.978269    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"
Nov 03 22:10:17.010990 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:17.009439    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-kube-api-access-rr2fl\" (UniqueName: \"kubernetes.io/projected/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-kube-api-access-rr2fl\") pod \"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225\" (UID: \"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225\") "
Nov 03 22:10:17.010990 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:17.009514    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-openshift-service-ca\") pod \"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225\" (UID: \"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225\") "
Nov 03 22:10:17.010990 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:17.009731    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-maas-api-tls\") pod \"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225\" (UID: \"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225\") "
Nov 03 22:10:17.010990 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:17.009882    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-metrics-certs\") pod \"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225\" (UID: \"ce0ed29f-9f1e-4ac2-80d5-b806fa23a225\") "
Nov 03 22:10:17.012295 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:17.011955    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-openshift-service-ca" pod "ce0ed29f-9f1e-4ac2-80d5-b806fa23a225" (UID: "ce0ed29f-9f1e-4ac2-80d5-b806fa23a225"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:10:17.013681 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:17.013641    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-metrics-certs" pod "ce0ed29f-9f1e-4ac2-80d5-b806fa23a225" (UID: "ce0ed29f-9f1e-4ac2-80d5-b806fa23a225"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:10:17.015893 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:17.015869    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-kube-api-access-rr2fl" pod "ce0ed29f-9f1e-4ac2-80d5-b806fa23a225" (UID: "ce0ed29f-9f1e-4ac2-80d5-b806fa23a225"). InnerVolumeSpecName "kube-api-access-rr2fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:10:17.016921 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:17.016893    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-maas-api-tls" pod "ce0ed29f-9f1e-4ac2-80d5-b806fa23a225" (UID: "ce0ed29f-9f1e-4ac2-80d5-b806fa23a225"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:10:17.057862 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:17.057831    5053 scope.go:122] "RemoveContainer" containerID="ede53729f16145eeebcb079a9f48bc0d716a5a2d9170a43615c543c8de74a9fc"
Nov 03 22:10:17.058427 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:10:17.058358    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede53729f16145eeebcb079a9f48bc0d716a5a2d9170a43615c543c8de74a9fc\": container with ID starting with ede53729f16145eeebcb079a9f48bc0d716a5a2d9170a43615c543c8de74a9fc not found: ID does not exist" containerID="ede53729f16145eeebcb079a9f48bc0d716a5a2d9170a43615c543c8de74a9fc"
Nov 03 22:10:17.058427 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:17.058405    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede53729f16145eeebcb079a9f48bc0d716a5a2d9170a43615c543c8de74a9fc"} err="failed to get container status \"ede53729f16145eeebcb079a9f48bc0d716a5a2d9170a43615c543c8de74a9fc\": rpc error: code = NotFound desc = could not find container \"ede53729f16145eeebcb079a9f48bc0d716a5a2d9170a43615c543c8de74a9fc\": container with ID starting with ede53729f16145eeebcb079a9f48bc0d716a5a2d9170a43615c543c8de74a9fc not found: ID does not exist"
Nov 03 22:10:17.111741 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:17.111701    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr2fl\" (UniqueName: \"kubernetes.io/projected/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-kube-api-access-rr2fl\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:17.111741 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:17.111733    5053 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-openshift-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:17.111741 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:17.111744    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-maas-api-tls\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:17.111741 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:17.111753    5053 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225-metrics-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:17.399619 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:17.399583    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"]
Nov 03 22:10:17.404006 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:17.403978    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-27a3c3-545f76647f-x9dxn"]
Nov 03 22:10:17.668950 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:17.668867    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce0ed29f-9f1e-4ac2-80d5-b806fa23a225" path="/var/lib/kubelet/pods/ce0ed29f-9f1e-4ac2-80d5-b806fa23a225/volumes"
Nov 03 22:10:23.046123 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:23.046085    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac97e0aa-q6dzz"]
Nov 03 22:10:23.095901 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:23.095867    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac97e0aa-q6dzz"]
Nov 03 22:10:23.672187 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:23.672150    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cf62825-a342-46ec-af65-00b81a746e6a" path="/var/lib/kubelet/pods/9cf62825-a342-46ec-af65-00b81a746e6a/volumes"
Nov 03 22:10:25.589882 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:25.589842    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"]
Nov 03 22:10:25.590418 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:25.590297    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv" podUID="af213d85-c84b-4ae2-b533-77c12c615a96" containerName="istio-proxy" containerID="cri-o://3e36aebae6007cd474d5192428752a149de8140293faf0d7b6c918af509db9fd" gracePeriod=30
Nov 03 22:10:26.851411 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:26.851378    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"]
Nov 03 22:10:26.852052 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:26.852037    5053 cpu_manager.go:414] "containerMap: removing container" podUID="c35172ca-45a9-42da-a7dc-645dad791743" containerName="test-curl-7676-e1f049"
Nov 03 22:10:26.852112 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:26.852053    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c35172ca-45a9-42da-a7dc-645dad791743" containerName="test-curl-7676-e1f049"
Nov 03 22:10:26.852112 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:26.852072    5053 cpu_manager.go:414] "containerMap: removing container" podUID="ce0ed29f-9f1e-4ac2-80d5-b806fa23a225" containerName="maas-api"
Nov 03 22:10:26.852112 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:26.852078    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce0ed29f-9f1e-4ac2-80d5-b806fa23a225" containerName="maas-api"
Nov 03 22:10:26.852215 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:26.852195    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="c35172ca-45a9-42da-a7dc-645dad791743" containerName="test-curl-7676-e1f049"
Nov 03 22:10:26.852215 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:26.852206    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce0ed29f-9f1e-4ac2-80d5-b806fa23a225" containerName="maas-api"
Nov 03 22:10:26.857634 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:26.857610    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:26.923982 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:26.923944    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"]
Nov 03 22:10:26.959896 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:26.959855    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvkbm\" (UniqueName: \"kubernetes.io/projected/0a6f0aba-49c7-4dc2-8616-38aa09ead824-kube-api-access-gvkbm\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:26.960115 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:26.959936    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-credential-socket\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:26.960115 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:26.960059    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istiod-ca-cert\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:26.960115 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:26.960093    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-workload-certs\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:26.960385 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:26.960166    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-token\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:26.960385 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:26.960211    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-envoy\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:26.960385 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:26.960291    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-podinfo\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:26.960385 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:26.960352    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-workload-socket\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:26.960627 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:26.960488    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-data\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:27.015267 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:27.007840    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-79e7c7-openshift-default-dockercfg-lssxf\""
Nov 03 22:10:27.062065 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:27.062011    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-data\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:27.062852 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:27.062815    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvkbm\" (UniqueName: \"kubernetes.io/projected/0a6f0aba-49c7-4dc2-8616-38aa09ead824-kube-api-access-gvkbm\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:27.063475 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:27.063452    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-credential-socket\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:27.064416 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:27.064396    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istiod-ca-cert\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:27.068389 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:27.068349    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-workload-certs\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:27.069541 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:27.068968    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-token\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:27.069731 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:27.069713    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-envoy\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:27.069856 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:27.069841    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-podinfo\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:27.070317 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:27.070296    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-workload-socket\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:27.070530 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:27.062731    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-data\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:27.070933 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:27.063870    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-credential-socket\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:27.071117 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:27.068816    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-workload-certs\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:27.071211 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:27.068871    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istiod-ca-cert\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:27.071367 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:27.070879    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-workload-socket\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:27.083289 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:27.075756    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-podinfo\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:27.090054 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:27.090020    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-envoy\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:27.120051 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:27.119973    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvkbm\" (UniqueName: \"kubernetes.io/projected/0a6f0aba-49c7-4dc2-8616-38aa09ead824-kube-api-access-gvkbm\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:27.125696 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:27.125669    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-token\") pod \"e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") " pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:27.171564 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:27.171529    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:27.609130 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:27.609094    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"]
Nov 03 22:10:27.648339 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:10:27.648300    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a6f0aba_49c7_4dc2_8616_38aa09ead824.slice/crio-8d04ad341f2dec91cecbf12e61cf8d5ef23fa249dc3840fc131b8c5d28a64a1f WatchSource:0}: Error finding container 8d04ad341f2dec91cecbf12e61cf8d5ef23fa249dc3840fc131b8c5d28a64a1f: Status 404 returned error can't find the container with id 8d04ad341f2dec91cecbf12e61cf8d5ef23fa249dc3840fc131b8c5d28a64a1f
Nov 03 22:10:27.651441 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:27.651407    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:10:27.651552 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:27.651476    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:10:27.651552 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:27.651507    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:10:28.088025 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:28.087989    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l" event={"ID":"0a6f0aba-49c7-4dc2-8616-38aa09ead824","Type":"ContainerStarted","Data":"2460323dd6f3c2d88450279c4d390b76c7dfbf7f62bd7549e370733d6a1a0b77"}
Nov 03 22:10:28.088552 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:28.088057    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l" event={"ID":"0a6f0aba-49c7-4dc2-8616-38aa09ead824","Type":"ContainerStarted","Data":"8d04ad341f2dec91cecbf12e61cf8d5ef23fa249dc3840fc131b8c5d28a64a1f"}
Nov 03 22:10:28.172616 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:28.172582    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:28.179396 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:28.179367    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:28.204305 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:28.204250    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l" podStartSLOduration=2.204220503 podStartE2EDuration="2.204220503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:10:28.203909795 +0000 UTC m=+3342.697831715" watchObservedRunningTime="2026-11-03 22:10:28.204220503 +0000 UTC m=+3342.698142378"
Nov 03 22:10:29.094834 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:29.094805    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:29.095926 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:29.095900    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:10:30.886817 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:30.886791    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:10:30.936044 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:30.936013    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af213d85-c84b-4ae2-b533-77c12c615a96-istio-token\" (UniqueName: \"kubernetes.io/projected/af213d85-c84b-4ae2-b533-77c12c615a96-istio-token\") pod \"af213d85-c84b-4ae2-b533-77c12c615a96\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") "
Nov 03 22:10:30.936210 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:30.936059    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af213d85-c84b-4ae2-b533-77c12c615a96-kube-api-access-g4bhd\" (UniqueName: \"kubernetes.io/projected/af213d85-c84b-4ae2-b533-77c12c615a96-kube-api-access-g4bhd\") pod \"af213d85-c84b-4ae2-b533-77c12c615a96\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") "
Nov 03 22:10:30.936210 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:30.936173    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af213d85-c84b-4ae2-b533-77c12c615a96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af213d85-c84b-4ae2-b533-77c12c615a96-istio-podinfo\") pod \"af213d85-c84b-4ae2-b533-77c12c615a96\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") "
Nov 03 22:10:30.936354 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:30.936312    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-credential-socket\") pod \"af213d85-c84b-4ae2-b533-77c12c615a96\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") "
Nov 03 22:10:30.936416 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:30.936379    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-istio-data\") pod \"af213d85-c84b-4ae2-b533-77c12c615a96\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") "
Nov 03 22:10:30.936482 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:30.936436    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-istio-envoy\") pod \"af213d85-c84b-4ae2-b533-77c12c615a96\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") "
Nov 03 22:10:30.936547 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:30.936489    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-workload-certs\") pod \"af213d85-c84b-4ae2-b533-77c12c615a96\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") "
Nov 03 22:10:30.936547 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:30.936540    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-workload-socket\") pod \"af213d85-c84b-4ae2-b533-77c12c615a96\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") "
Nov 03 22:10:30.936660 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:30.936583    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af213d85-c84b-4ae2-b533-77c12c615a96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af213d85-c84b-4ae2-b533-77c12c615a96-istiod-ca-cert\") pod \"af213d85-c84b-4ae2-b533-77c12c615a96\" (UID: \"af213d85-c84b-4ae2-b533-77c12c615a96\") "
Nov 03 22:10:30.936660 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:30.936618    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-credential-socket" pod "af213d85-c84b-4ae2-b533-77c12c615a96" (UID: "af213d85-c84b-4ae2-b533-77c12c615a96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:10:30.936872 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:30.936848    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-workload-socket" pod "af213d85-c84b-4ae2-b533-77c12c615a96" (UID: "af213d85-c84b-4ae2-b533-77c12c615a96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:10:30.937096 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:30.937070    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-workload-certs" pod "af213d85-c84b-4ae2-b533-77c12c615a96" (UID: "af213d85-c84b-4ae2-b533-77c12c615a96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:10:30.937511 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:30.937488    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af213d85-c84b-4ae2-b533-77c12c615a96-istiod-ca-cert" pod "af213d85-c84b-4ae2-b533-77c12c615a96" (UID: "af213d85-c84b-4ae2-b533-77c12c615a96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:10:30.937669 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:30.937649    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-istio-data" pod "af213d85-c84b-4ae2-b533-77c12c615a96" (UID: "af213d85-c84b-4ae2-b533-77c12c615a96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:10:30.938311 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:30.938231    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:30.938413 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:30.938320    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:30.938502 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:30.938467    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:30.938502 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:30.938486    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af213d85-c84b-4ae2-b533-77c12c615a96-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:30.938618 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:30.938502    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:30.939002 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:30.938978    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af213d85-c84b-4ae2-b533-77c12c615a96-kube-api-access-g4bhd" pod "af213d85-c84b-4ae2-b533-77c12c615a96" (UID: "af213d85-c84b-4ae2-b533-77c12c615a96"). InnerVolumeSpecName "kube-api-access-g4bhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:10:30.939090 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:30.939015    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af213d85-c84b-4ae2-b533-77c12c615a96-istio-token" pod "af213d85-c84b-4ae2-b533-77c12c615a96" (UID: "af213d85-c84b-4ae2-b533-77c12c615a96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:10:30.939306 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:30.939283    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-istio-envoy" pod "af213d85-c84b-4ae2-b533-77c12c615a96" (UID: "af213d85-c84b-4ae2-b533-77c12c615a96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:10:30.939381 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:30.939285    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af213d85-c84b-4ae2-b533-77c12c615a96-istio-podinfo" pod "af213d85-c84b-4ae2-b533-77c12c615a96" (UID: "af213d85-c84b-4ae2-b533-77c12c615a96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:10:31.039618 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:31.039584    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af213d85-c84b-4ae2-b533-77c12c615a96-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:31.039618 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:31.039618    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af213d85-c84b-4ae2-b533-77c12c615a96-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:31.039834 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:31.039634    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4bhd\" (UniqueName: \"kubernetes.io/projected/af213d85-c84b-4ae2-b533-77c12c615a96-kube-api-access-g4bhd\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:31.039834 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:31.039649    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af213d85-c84b-4ae2-b533-77c12c615a96-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:31.111508 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:31.111415    5053 generic.go:356] "Generic (PLEG): container finished" podID="af213d85-c84b-4ae2-b533-77c12c615a96" containerID="3e36aebae6007cd474d5192428752a149de8140293faf0d7b6c918af509db9fd" exitCode=0
Nov 03 22:10:31.111508 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:31.111481    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"
Nov 03 22:10:31.111769 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:31.111500    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv" event={"ID":"af213d85-c84b-4ae2-b533-77c12c615a96","Type":"ContainerDied","Data":"3e36aebae6007cd474d5192428752a149de8140293faf0d7b6c918af509db9fd"}
Nov 03 22:10:31.111769 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:31.111569    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv" event={"ID":"af213d85-c84b-4ae2-b533-77c12c615a96","Type":"ContainerDied","Data":"e4f04727252ca25b6129f5de7db67830b39f860cd9a41749a7df29b5dc3a6ba8"}
Nov 03 22:10:31.111769 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:31.111601    5053 scope.go:122] "RemoveContainer" containerID="3e36aebae6007cd474d5192428752a149de8140293faf0d7b6c918af509db9fd"
Nov 03 22:10:31.123547 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:31.123526    5053 scope.go:122] "RemoveContainer" containerID="76e2a573d15add655d1d5f6c3908d836cb930d51fd03c35ceef81f90e4024f10"
Nov 03 22:10:31.133648 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:31.133630    5053 scope.go:122] "RemoveContainer" containerID="3e36aebae6007cd474d5192428752a149de8140293faf0d7b6c918af509db9fd"
Nov 03 22:10:31.133938 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:10:31.133917    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e36aebae6007cd474d5192428752a149de8140293faf0d7b6c918af509db9fd\": container with ID starting with 3e36aebae6007cd474d5192428752a149de8140293faf0d7b6c918af509db9fd not found: ID does not exist" containerID="3e36aebae6007cd474d5192428752a149de8140293faf0d7b6c918af509db9fd"
Nov 03 22:10:31.134018 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:31.133949    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e36aebae6007cd474d5192428752a149de8140293faf0d7b6c918af509db9fd"} err="failed to get container status \"3e36aebae6007cd474d5192428752a149de8140293faf0d7b6c918af509db9fd\": rpc error: code = NotFound desc = could not find container \"3e36aebae6007cd474d5192428752a149de8140293faf0d7b6c918af509db9fd\": container with ID starting with 3e36aebae6007cd474d5192428752a149de8140293faf0d7b6c918af509db9fd not found: ID does not exist"
Nov 03 22:10:31.134018 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:31.133966    5053 scope.go:122] "RemoveContainer" containerID="76e2a573d15add655d1d5f6c3908d836cb930d51fd03c35ceef81f90e4024f10"
Nov 03 22:10:31.134332 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:10:31.134225    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76e2a573d15add655d1d5f6c3908d836cb930d51fd03c35ceef81f90e4024f10\": container with ID starting with 76e2a573d15add655d1d5f6c3908d836cb930d51fd03c35ceef81f90e4024f10 not found: ID does not exist" containerID="76e2a573d15add655d1d5f6c3908d836cb930d51fd03c35ceef81f90e4024f10"
Nov 03 22:10:31.134414 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:31.134344    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e2a573d15add655d1d5f6c3908d836cb930d51fd03c35ceef81f90e4024f10"} err="failed to get container status \"76e2a573d15add655d1d5f6c3908d836cb930d51fd03c35ceef81f90e4024f10\": rpc error: code = NotFound desc = could not find container \"76e2a573d15add655d1d5f6c3908d836cb930d51fd03c35ceef81f90e4024f10\": container with ID starting with 76e2a573d15add655d1d5f6c3908d836cb930d51fd03c35ceef81f90e4024f10 not found: ID does not exist"
Nov 03 22:10:31.275708 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:31.275677    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"]
Nov 03 22:10:31.279853 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:31.279828    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-27a3c3-openshift-default-5d58b7857c-xhgnv"]
Nov 03 22:10:31.667623 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:31.667591    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af213d85-c84b-4ae2-b533-77c12c615a96" path="/var/lib/kubelet/pods/af213d85-c84b-4ae2-b533-77c12c615a96/volumes"
Nov 03 22:10:35.370222 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:35.370192    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"]
Nov 03 22:10:40.929257 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:10:40.929196    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:10:48.042648 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:48.042613    5053 scope.go:122] "RemoveContainer" containerID="d084b1000efa923450ba423af869c04b5365a8b9408265b8ea715081ef0f7e85"
Nov 03 22:10:50.211087 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.211051    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"]
Nov 03 22:10:50.211771 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.211753    5053 cpu_manager.go:414] "containerMap: removing container" podUID="af213d85-c84b-4ae2-b533-77c12c615a96" containerName="istio-proxy"
Nov 03 22:10:50.211771 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.211773    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af213d85-c84b-4ae2-b533-77c12c615a96" containerName="istio-proxy"
Nov 03 22:10:50.211904 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.211803    5053 cpu_manager.go:414] "containerMap: removing container" podUID="af213d85-c84b-4ae2-b533-77c12c615a96" containerName="istio-proxy"
Nov 03 22:10:50.211904 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.211809    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af213d85-c84b-4ae2-b533-77c12c615a96" containerName="istio-proxy"
Nov 03 22:10:50.211974 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.211916    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="af213d85-c84b-4ae2-b533-77c12c615a96" containerName="istio-proxy"
Nov 03 22:10:50.216849 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.216831    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.307738 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.307676    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-12dee4-openshift-default-dockercfg-n6v9k\""
Nov 03 22:10:50.310320 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.310274    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-istio-data\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.310615 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.310583    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-workload-socket\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.310788 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.310768    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6862760e-4710-4dc1-afd1-30df93440970-istio-token\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.310999 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.310969    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-istio-envoy\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.311133 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.311116    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-credential-socket\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.311306 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.311289    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8pb9\" (UniqueName: \"kubernetes.io/projected/6862760e-4710-4dc1-afd1-30df93440970-kube-api-access-p8pb9\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.311586 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.311563    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6862760e-4710-4dc1-afd1-30df93440970-istiod-ca-cert\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.311752 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.311734    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6862760e-4710-4dc1-afd1-30df93440970-istio-podinfo\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.311944 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.311924    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-workload-certs\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.396654 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.395314    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd" podUID="225eed60-7bae-455a-baac-3c013d12a51c" containerName="main" containerID="cri-o://0c714f7ac0ee0f79ea2ac9746ec9a2289075f5d19db5e0cabf657d74b0abd7a0" gracePeriod=45
Nov 03 22:10:50.424289 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.421999    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8pb9\" (UniqueName: \"kubernetes.io/projected/6862760e-4710-4dc1-afd1-30df93440970-kube-api-access-p8pb9\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.424289 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.422083    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6862760e-4710-4dc1-afd1-30df93440970-istiod-ca-cert\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.424289 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.422117    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6862760e-4710-4dc1-afd1-30df93440970-istio-podinfo\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.424289 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.422206    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-workload-certs\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.424289 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.422382    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-istio-data\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.424289 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.422437    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-workload-socket\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.424289 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.422457    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6862760e-4710-4dc1-afd1-30df93440970-istio-token\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.424289 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.422488    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-istio-envoy\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.424289 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.422535    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-credential-socket\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.424289 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.422971    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-credential-socket\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.424289 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.423935    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6862760e-4710-4dc1-afd1-30df93440970-istiod-ca-cert\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.425070 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.424607    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-workload-socket\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.425070 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.424870    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-workload-certs\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.425195 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.425107    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-istio-data\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.428838 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.428813    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-istio-envoy\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.435937 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.435900    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6862760e-4710-4dc1-afd1-30df93440970-istio-podinfo\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.477749 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.477668    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"]
Nov 03 22:10:50.480555 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.480496    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8pb9\" (UniqueName: \"kubernetes.io/projected/6862760e-4710-4dc1-afd1-30df93440970-kube-api-access-p8pb9\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.482036 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.481985    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6862760e-4710-4dc1-afd1-30df93440970-istio-token\") pod \"e2e-autores-12dee4-openshift-default-577996447c-jqml4\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") " pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.553302 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.553263    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:50.734201 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.733571    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:10:50.832149 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.832110    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-model-cache\") pod \"225eed60-7bae-455a-baac-3c013d12a51c\" (UID: \"225eed60-7bae-455a-baac-3c013d12a51c\") "
Nov 03 22:10:50.832149 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.832158    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/225eed60-7bae-455a-baac-3c013d12a51c-kube-api-access-xkrrk\" (UniqueName: \"kubernetes.io/projected/225eed60-7bae-455a-baac-3c013d12a51c-kube-api-access-xkrrk\") pod \"225eed60-7bae-455a-baac-3c013d12a51c\" (UID: \"225eed60-7bae-455a-baac-3c013d12a51c\") "
Nov 03 22:10:50.832512 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.832483    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-model-cache" pod "225eed60-7bae-455a-baac-3c013d12a51c" (UID: "225eed60-7bae-455a-baac-3c013d12a51c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:10:50.832606 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.832584    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/225eed60-7bae-455a-baac-3c013d12a51c-tls-certs\" (UniqueName: \"kubernetes.io/secret/225eed60-7bae-455a-baac-3c013d12a51c-tls-certs\") pod \"225eed60-7bae-455a-baac-3c013d12a51c\" (UID: \"225eed60-7bae-455a-baac-3c013d12a51c\") "
Nov 03 22:10:50.832675 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.832629    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-tmp-dir\") pod \"225eed60-7bae-455a-baac-3c013d12a51c\" (UID: \"225eed60-7bae-455a-baac-3c013d12a51c\") "
Nov 03 22:10:50.832964 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.832795    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-home\" (UniqueName: \"kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-home\") pod \"225eed60-7bae-455a-baac-3c013d12a51c\" (UID: \"225eed60-7bae-455a-baac-3c013d12a51c\") "
Nov 03 22:10:50.832964 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.832908    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-dshm\") pod \"225eed60-7bae-455a-baac-3c013d12a51c\" (UID: \"225eed60-7bae-455a-baac-3c013d12a51c\") "
Nov 03 22:10:50.833358 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.833018    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-tmp-dir" pod "225eed60-7bae-455a-baac-3c013d12a51c" (UID: "225eed60-7bae-455a-baac-3c013d12a51c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:10:50.833358 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.833114    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-home" pod "225eed60-7bae-455a-baac-3c013d12a51c" (UID: "225eed60-7bae-455a-baac-3c013d12a51c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:10:50.833788 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.833754    5053 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-tmp-dir\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:50.833788 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.833783    5053 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-home\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:50.833944 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.833797    5053 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-model-cache\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:50.835099 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.835063    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/225eed60-7bae-455a-baac-3c013d12a51c-tls-certs" pod "225eed60-7bae-455a-baac-3c013d12a51c" (UID: "225eed60-7bae-455a-baac-3c013d12a51c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:10:50.836433 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.836405    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/225eed60-7bae-455a-baac-3c013d12a51c-kube-api-access-xkrrk" pod "225eed60-7bae-455a-baac-3c013d12a51c" (UID: "225eed60-7bae-455a-baac-3c013d12a51c"). InnerVolumeSpecName "kube-api-access-xkrrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:10:50.837224 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.837194    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-dshm" pod "225eed60-7bae-455a-baac-3c013d12a51c" (UID: "225eed60-7bae-455a-baac-3c013d12a51c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:10:50.935382 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.935340    5053 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/225eed60-7bae-455a-baac-3c013d12a51c-dshm\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:50.935382 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.935382    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkrrk\" (UniqueName: \"kubernetes.io/projected/225eed60-7bae-455a-baac-3c013d12a51c-kube-api-access-xkrrk\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:50.935382 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:50.935396    5053 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/225eed60-7bae-455a-baac-3c013d12a51c-tls-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:51.010660 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:51.010631    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"]
Nov 03 22:10:51.012639 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:10:51.012600    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6862760e_4710_4dc1_afd1_30df93440970.slice/crio-1f84b8881584637138181e0626b05ceb0611854197c9c14c7206b2e9cff4ad36 WatchSource:0}: Error finding container 1f84b8881584637138181e0626b05ceb0611854197c9c14c7206b2e9cff4ad36: Status 404 returned error can't find the container with id 1f84b8881584637138181e0626b05ceb0611854197c9c14c7206b2e9cff4ad36
Nov 03 22:10:51.015781 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:51.015737    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:10:51.015849 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:51.015832    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:10:51.015891 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:51.015874    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:10:51.405408 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:51.405298    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4" event={"ID":"6862760e-4710-4dc1-afd1-30df93440970","Type":"ContainerStarted","Data":"7e251468046f1aa4bec1d2389207a5003941723afbc5f1f63f051afecf56d636"}
Nov 03 22:10:51.405408 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:51.405364    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4" event={"ID":"6862760e-4710-4dc1-afd1-30df93440970","Type":"ContainerStarted","Data":"1f84b8881584637138181e0626b05ceb0611854197c9c14c7206b2e9cff4ad36"}
Nov 03 22:10:51.407146 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:51.407112    5053 generic.go:356] "Generic (PLEG): container finished" podID="225eed60-7bae-455a-baac-3c013d12a51c" containerID="0c714f7ac0ee0f79ea2ac9746ec9a2289075f5d19db5e0cabf657d74b0abd7a0" exitCode=0
Nov 03 22:10:51.407364 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:51.407149    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd" event={"ID":"225eed60-7bae-455a-baac-3c013d12a51c","Type":"ContainerDied","Data":"0c714f7ac0ee0f79ea2ac9746ec9a2289075f5d19db5e0cabf657d74b0abd7a0"}
Nov 03 22:10:51.407364 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:51.407177    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"
Nov 03 22:10:51.407364 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:51.407196    5053 scope.go:122] "RemoveContainer" containerID="0c714f7ac0ee0f79ea2ac9746ec9a2289075f5d19db5e0cabf657d74b0abd7a0"
Nov 03 22:10:51.407364 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:51.407184    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd" event={"ID":"225eed60-7bae-455a-baac-3c013d12a51c","Type":"ContainerDied","Data":"d610bfc5ee901918b9ec612ef4b2842306bb6ed0679f21fa9f55679e80610f30"}
Nov 03 22:10:51.439779 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:51.439759    5053 scope.go:122] "RemoveContainer" containerID="0c714f7ac0ee0f79ea2ac9746ec9a2289075f5d19db5e0cabf657d74b0abd7a0"
Nov 03 22:10:51.440151 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:10:51.440122    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c714f7ac0ee0f79ea2ac9746ec9a2289075f5d19db5e0cabf657d74b0abd7a0\": container with ID starting with 0c714f7ac0ee0f79ea2ac9746ec9a2289075f5d19db5e0cabf657d74b0abd7a0 not found: ID does not exist" containerID="0c714f7ac0ee0f79ea2ac9746ec9a2289075f5d19db5e0cabf657d74b0abd7a0"
Nov 03 22:10:51.440286 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:51.440162    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c714f7ac0ee0f79ea2ac9746ec9a2289075f5d19db5e0cabf657d74b0abd7a0"} err="failed to get container status \"0c714f7ac0ee0f79ea2ac9746ec9a2289075f5d19db5e0cabf657d74b0abd7a0\": rpc error: code = NotFound desc = could not find container \"0c714f7ac0ee0f79ea2ac9746ec9a2289075f5d19db5e0cabf657d74b0abd7a0\": container with ID starting with 0c714f7ac0ee0f79ea2ac9746ec9a2289075f5d19db5e0cabf657d74b0abd7a0 not found: ID does not exist"
Nov 03 22:10:51.521332 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:51.521267    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4" podStartSLOduration=1.521223069 podStartE2EDuration="1.521223069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:10:51.520548733 +0000 UTC m=+3366.014470609" watchObservedRunningTime="2026-11-03 22:10:51.521223069 +0000 UTC m=+3366.015144946"
Nov 03 22:10:51.554027 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:51.553996    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:51.639263 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:51.637184    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"]
Nov 03 22:10:51.642072 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:51.642042    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-87e1cf/auth-test-model-87e1cf-kserve-79d84d5d9-jxbkd"]
Nov 03 22:10:51.674084 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:51.674004    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="225eed60-7bae-455a-baac-3c013d12a51c" path="/var/lib/kubelet/pods/225eed60-7bae-455a-baac-3c013d12a51c/volumes"
Nov 03 22:10:52.560203 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:52.560172    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:53.426055 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:53.426022    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:10:53.427148 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:10:53.427125    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:11:00.423973 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.423938    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"]
Nov 03 22:11:00.424636 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.424619    5053 cpu_manager.go:414] "containerMap: removing container" podUID="225eed60-7bae-455a-baac-3c013d12a51c" containerName="main"
Nov 03 22:11:00.424682 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.424637    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="225eed60-7bae-455a-baac-3c013d12a51c" containerName="main"
Nov 03 22:11:00.424751 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.424742    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="225eed60-7bae-455a-baac-3c013d12a51c" containerName="main"
Nov 03 22:11:00.424784 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.424756    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="af213d85-c84b-4ae2-b533-77c12c615a96" containerName="istio-proxy"
Nov 03 22:11:00.429504 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.429488    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:11:00.487111 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.487068    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"]
Nov 03 22:11:00.487730 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.487704    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-home\") pod \"noten-model-79e7c7-kserve-b48b6dbb9-f9w9l\" (UID: \"ad53efb5-e967-4315-8585-56ff322bb2b3\") " pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:11:00.487812 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.487742    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdlb5\" (UniqueName: \"kubernetes.io/projected/ad53efb5-e967-4315-8585-56ff322bb2b3-kube-api-access-qdlb5\") pod \"noten-model-79e7c7-kserve-b48b6dbb9-f9w9l\" (UID: \"ad53efb5-e967-4315-8585-56ff322bb2b3\") " pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:11:00.487863 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.487809    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-model-cache\") pod \"noten-model-79e7c7-kserve-b48b6dbb9-f9w9l\" (UID: \"ad53efb5-e967-4315-8585-56ff322bb2b3\") " pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:11:00.487863 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.487832    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-tmp-dir\") pod \"noten-model-79e7c7-kserve-b48b6dbb9-f9w9l\" (UID: \"ad53efb5-e967-4315-8585-56ff322bb2b3\") " pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:11:00.487936 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.487897    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-dshm\") pod \"noten-model-79e7c7-kserve-b48b6dbb9-f9w9l\" (UID: \"ad53efb5-e967-4315-8585-56ff322bb2b3\") " pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:11:00.487936 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.487912    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad53efb5-e967-4315-8585-56ff322bb2b3-tls-certs\") pod \"noten-model-79e7c7-kserve-b48b6dbb9-f9w9l\" (UID: \"ad53efb5-e967-4315-8585-56ff322bb2b3\") " pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:11:00.531442 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.531393    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-79e7c7\"/\"default-dockercfg-brm4r\""
Nov 03 22:11:00.531442 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.531419    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-79e7c7\"/\"openshift-service-ca.crt\""
Nov 03 22:11:00.531773 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.531705    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-79e7c7\"/\"noten-model-79e7c7-kserve-self-signed-certs\""
Nov 03 22:11:00.531773 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.531715    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-79e7c7\"/\"kube-root-ca.crt\""
Nov 03 22:11:00.589571 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.589538    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad53efb5-e967-4315-8585-56ff322bb2b3-tls-certs\") pod \"noten-model-79e7c7-kserve-b48b6dbb9-f9w9l\" (UID: \"ad53efb5-e967-4315-8585-56ff322bb2b3\") " pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:11:00.589806 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.589691    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-home\") pod \"noten-model-79e7c7-kserve-b48b6dbb9-f9w9l\" (UID: \"ad53efb5-e967-4315-8585-56ff322bb2b3\") " pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:11:00.589806 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.589713    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdlb5\" (UniqueName: \"kubernetes.io/projected/ad53efb5-e967-4315-8585-56ff322bb2b3-kube-api-access-qdlb5\") pod \"noten-model-79e7c7-kserve-b48b6dbb9-f9w9l\" (UID: \"ad53efb5-e967-4315-8585-56ff322bb2b3\") " pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:11:00.589806 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.589776    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-model-cache\") pod \"noten-model-79e7c7-kserve-b48b6dbb9-f9w9l\" (UID: \"ad53efb5-e967-4315-8585-56ff322bb2b3\") " pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:11:00.589806 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.589799    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-tmp-dir\") pod \"noten-model-79e7c7-kserve-b48b6dbb9-f9w9l\" (UID: \"ad53efb5-e967-4315-8585-56ff322bb2b3\") " pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:11:00.590052 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.589862    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-dshm\") pod \"noten-model-79e7c7-kserve-b48b6dbb9-f9w9l\" (UID: \"ad53efb5-e967-4315-8585-56ff322bb2b3\") " pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:11:00.591162 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.590776    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-home\") pod \"noten-model-79e7c7-kserve-b48b6dbb9-f9w9l\" (UID: \"ad53efb5-e967-4315-8585-56ff322bb2b3\") " pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:11:00.591531 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.591505    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-tmp-dir\") pod \"noten-model-79e7c7-kserve-b48b6dbb9-f9w9l\" (UID: \"ad53efb5-e967-4315-8585-56ff322bb2b3\") " pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:11:00.591852 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.591828    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-model-cache\") pod \"noten-model-79e7c7-kserve-b48b6dbb9-f9w9l\" (UID: \"ad53efb5-e967-4315-8585-56ff322bb2b3\") " pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:11:00.593945 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.593919    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-dshm\") pod \"noten-model-79e7c7-kserve-b48b6dbb9-f9w9l\" (UID: \"ad53efb5-e967-4315-8585-56ff322bb2b3\") " pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:11:00.597972 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.597952    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad53efb5-e967-4315-8585-56ff322bb2b3-tls-certs\") pod \"noten-model-79e7c7-kserve-b48b6dbb9-f9w9l\" (UID: \"ad53efb5-e967-4315-8585-56ff322bb2b3\") " pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:11:00.646589 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.646555    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdlb5\" (UniqueName: \"kubernetes.io/projected/ad53efb5-e967-4315-8585-56ff322bb2b3-kube-api-access-qdlb5\") pod \"noten-model-79e7c7-kserve-b48b6dbb9-f9w9l\" (UID: \"ad53efb5-e967-4315-8585-56ff322bb2b3\") " pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:11:00.743825 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:00.743786    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:11:01.123478 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:01.123450    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"]
Nov 03 22:11:01.124811 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:11:01.124779    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad53efb5_e967_4315_8585_56ff322bb2b3.slice/crio-f8a5a800b153f7e3caf83263acf84ea08d6653737b6068015b9fc11eceed1942 WatchSource:0}: Error finding container f8a5a800b153f7e3caf83263acf84ea08d6653737b6068015b9fc11eceed1942: Status 404 returned error can't find the container with id f8a5a800b153f7e3caf83263acf84ea08d6653737b6068015b9fc11eceed1942
Nov 03 22:11:01.502365 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:01.502323    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l" event={"ID":"ad53efb5-e967-4315-8585-56ff322bb2b3","Type":"ContainerStarted","Data":"5713dcaa702180b9d526dfc4892ed16dcf625c6750c1560b9bd9d4aabc584cea"}
Nov 03 22:11:01.502365 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:01.502371    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l" event={"ID":"ad53efb5-e967-4315-8585-56ff322bb2b3","Type":"ContainerStarted","Data":"f8a5a800b153f7e3caf83263acf84ea08d6653737b6068015b9fc11eceed1942"}
Nov 03 22:11:01.616778 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:01.616711    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l" podStartSLOduration=1.616696267 podStartE2EDuration="1.616696267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:11:01.615548684 +0000 UTC m=+3376.109470560" watchObservedRunningTime="2026-11-03 22:11:01.616696267 +0000 UTC m=+3376.110618142"
Nov 03 22:11:01.744623 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:01.744588    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:11:10.743970 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:10.743935    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:11:10.762571 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:10.762537    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:11:10.930486 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:11:10.930447    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:11:11.587523 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:11.587490    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:11:35.358863 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:35.358826    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"]
Nov 03 22:11:35.359494 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:35.359024    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b" podUID="7ee6091e-7d0a-43b8-836e-9565f25ecc5b" containerName="istio-proxy" containerID="cri-o://defba414ce9da88bb2955f57004a36450d31176b874736a7dac967e61c069867" gracePeriod=30
Nov 03 22:11:38.775765 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:38.775723    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:11:38.776164 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:38.775781    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b" podUID="7ee6091e-7d0a-43b8-836e-9565f25ecc5b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:11:40.807333 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:40.807299    5053 generic.go:356] "Generic (PLEG): container finished" podID="7ee6091e-7d0a-43b8-836e-9565f25ecc5b" containerID="defba414ce9da88bb2955f57004a36450d31176b874736a7dac967e61c069867" exitCode=0
Nov 03 22:11:40.807756 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:40.807382    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b" event={"ID":"7ee6091e-7d0a-43b8-836e-9565f25ecc5b","Type":"ContainerDied","Data":"defba414ce9da88bb2955f57004a36450d31176b874736a7dac967e61c069867"}
Nov 03 22:11:40.931836 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:11:40.931793    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:11:41.049587 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.049549    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:11:41.140707 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.140617    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-data\") pod \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") "
Nov 03 22:11:41.140707 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.140657    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-envoy\") pod \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") "
Nov 03 22:11:41.140959 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.140719    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-workload-socket\") pod \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") "
Nov 03 22:11:41.140959 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.140794    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-credential-socket\") pod \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") "
Nov 03 22:11:41.140959 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.140839    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-workload-certs\") pod \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") "
Nov 03 22:11:41.140959 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.140876    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istiod-ca-cert\") pod \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") "
Nov 03 22:11:41.140959 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.140931    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-token\" (UniqueName: \"kubernetes.io/projected/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-token\") pod \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") "
Nov 03 22:11:41.140959 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.140943    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-data" pod "7ee6091e-7d0a-43b8-836e-9565f25ecc5b" (UID: "7ee6091e-7d0a-43b8-836e-9565f25ecc5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:11:41.141357 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.141029    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-podinfo\") pod \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") "
Nov 03 22:11:41.141357 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.141113    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-kube-api-access-t6t8m\" (UniqueName: \"kubernetes.io/projected/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-kube-api-access-t6t8m\") pod \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\" (UID: \"7ee6091e-7d0a-43b8-836e-9565f25ecc5b\") "
Nov 03 22:11:41.141357 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.141119    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-workload-socket" pod "7ee6091e-7d0a-43b8-836e-9565f25ecc5b" (UID: "7ee6091e-7d0a-43b8-836e-9565f25ecc5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:11:41.141357 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.141179    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-workload-certs" pod "7ee6091e-7d0a-43b8-836e-9565f25ecc5b" (UID: "7ee6091e-7d0a-43b8-836e-9565f25ecc5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:11:41.141585 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.141405    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istiod-ca-cert" pod "7ee6091e-7d0a-43b8-836e-9565f25ecc5b" (UID: "7ee6091e-7d0a-43b8-836e-9565f25ecc5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:11:41.141585 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.141476    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-credential-socket" pod "7ee6091e-7d0a-43b8-836e-9565f25ecc5b" (UID: "7ee6091e-7d0a-43b8-836e-9565f25ecc5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:11:41.141745 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.141725    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:41.141999 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.141748    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:41.141999 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.141763    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:41.141999 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.141776    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:41.141999 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.141788    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:41.143753 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.143718    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-podinfo" pod "7ee6091e-7d0a-43b8-836e-9565f25ecc5b" (UID: "7ee6091e-7d0a-43b8-836e-9565f25ecc5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:11:41.143898 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.143877    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-envoy" pod "7ee6091e-7d0a-43b8-836e-9565f25ecc5b" (UID: "7ee6091e-7d0a-43b8-836e-9565f25ecc5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:11:41.143995 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.143970    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-token" pod "7ee6091e-7d0a-43b8-836e-9565f25ecc5b" (UID: "7ee6091e-7d0a-43b8-836e-9565f25ecc5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:11:41.144134 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.144103    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-kube-api-access-t6t8m" pod "7ee6091e-7d0a-43b8-836e-9565f25ecc5b" (UID: "7ee6091e-7d0a-43b8-836e-9565f25ecc5b"). InnerVolumeSpecName "kube-api-access-t6t8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:11:41.242937 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.242903    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:41.242937 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.242939    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:41.243185 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.242957    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6t8m\" (UniqueName: \"kubernetes.io/projected/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-kube-api-access-t6t8m\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:41.243185 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.242972    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee6091e-7d0a-43b8-836e-9565f25ecc5b-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:41.815782 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.815752    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"
Nov 03 22:11:41.816335 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.815779    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b" event={"ID":"7ee6091e-7d0a-43b8-836e-9565f25ecc5b","Type":"ContainerDied","Data":"06ef4261f9ab3e82117582744e14bf288683e6b4eaf4e12b12c5d5d46ef43b2a"}
Nov 03 22:11:41.816335 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:41.815850    5053 scope.go:122] "RemoveContainer" containerID="defba414ce9da88bb2955f57004a36450d31176b874736a7dac967e61c069867"
Nov 03 22:11:42.031794 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:42.031760    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"]
Nov 03 22:11:42.080415 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:42.080378    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9jp2b"]
Nov 03 22:11:42.090645 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:42.090611    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"]
Nov 03 22:11:42.090892 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:42.090865    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l" podUID="0a6f0aba-49c7-4dc2-8616-38aa09ead824" containerName="istio-proxy" containerID="cri-o://2460323dd6f3c2d88450279c4d390b76c7dfbf7f62bd7549e370733d6a1a0b77" gracePeriod=30
Nov 03 22:11:43.669188 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:43.669154    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee6091e-7d0a-43b8-836e-9565f25ecc5b" path="/var/lib/kubelet/pods/7ee6091e-7d0a-43b8-836e-9565f25ecc5b/volumes"
Nov 03 22:11:44.098327 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:44.098288    5053 patch_prober.go:29] interesting pod/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:11:44.098522 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:44.098365    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l" podUID="0a6f0aba-49c7-4dc2-8616-38aa09ead824" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:11:47.386103 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.386073    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:11:47.419648 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.419612    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-workload-certs\") pod \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") "
Nov 03 22:11:47.419847 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.419668    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-podinfo\") pod \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") "
Nov 03 22:11:47.419847 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.419707    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-data\") pod \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") "
Nov 03 22:11:47.419847 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.419725    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istiod-ca-cert\") pod \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") "
Nov 03 22:11:47.419847 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.419746    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-credential-socket\") pod \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") "
Nov 03 22:11:47.419847 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.419768    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-token\" (UniqueName: \"kubernetes.io/projected/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-token\") pod \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") "
Nov 03 22:11:47.420155 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.419921    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a6f0aba-49c7-4dc2-8616-38aa09ead824-kube-api-access-gvkbm\" (UniqueName: \"kubernetes.io/projected/0a6f0aba-49c7-4dc2-8616-38aa09ead824-kube-api-access-gvkbm\") pod \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") "
Nov 03 22:11:47.420155 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.419976    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-workload-certs" pod "0a6f0aba-49c7-4dc2-8616-38aa09ead824" (UID: "0a6f0aba-49c7-4dc2-8616-38aa09ead824"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:11:47.420155 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.419993    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-envoy\") pod \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") "
Nov 03 22:11:47.420155 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.420065    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-data" pod "0a6f0aba-49c7-4dc2-8616-38aa09ead824" (UID: "0a6f0aba-49c7-4dc2-8616-38aa09ead824"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:11:47.420155 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.420104    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-workload-socket\") pod \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\" (UID: \"0a6f0aba-49c7-4dc2-8616-38aa09ead824\") "
Nov 03 22:11:47.420425 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.420219    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-credential-socket" pod "0a6f0aba-49c7-4dc2-8616-38aa09ead824" (UID: "0a6f0aba-49c7-4dc2-8616-38aa09ead824"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:11:47.420633 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.420590    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-workload-socket" pod "0a6f0aba-49c7-4dc2-8616-38aa09ead824" (UID: "0a6f0aba-49c7-4dc2-8616-38aa09ead824"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:11:47.420756 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.420732    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istiod-ca-cert" pod "0a6f0aba-49c7-4dc2-8616-38aa09ead824" (UID: "0a6f0aba-49c7-4dc2-8616-38aa09ead824"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:11:47.420968 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.420947    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:47.421059 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.420974    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:47.421059 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.420989    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:47.421059 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.421004    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:47.422486 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.422458    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-envoy" pod "0a6f0aba-49c7-4dc2-8616-38aa09ead824" (UID: "0a6f0aba-49c7-4dc2-8616-38aa09ead824"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:11:47.422819 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.422793    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-podinfo" pod "0a6f0aba-49c7-4dc2-8616-38aa09ead824" (UID: "0a6f0aba-49c7-4dc2-8616-38aa09ead824"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:11:47.422819 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.422803    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a6f0aba-49c7-4dc2-8616-38aa09ead824-kube-api-access-gvkbm" pod "0a6f0aba-49c7-4dc2-8616-38aa09ead824" (UID: "0a6f0aba-49c7-4dc2-8616-38aa09ead824"). InnerVolumeSpecName "kube-api-access-gvkbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:11:47.423180 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.423159    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-token" pod "0a6f0aba-49c7-4dc2-8616-38aa09ead824" (UID: "0a6f0aba-49c7-4dc2-8616-38aa09ead824"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:11:47.522192 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.522156    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:47.522192 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.522188    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:47.522192 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.522197    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:47.522476 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.522208    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a6f0aba-49c7-4dc2-8616-38aa09ead824-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:47.522476 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.522216    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvkbm\" (UniqueName: \"kubernetes.io/projected/0a6f0aba-49c7-4dc2-8616-38aa09ead824-kube-api-access-gvkbm\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:47.615547 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.615444    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"]
Nov 03 22:11:47.865318 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.865273    5053 generic.go:356] "Generic (PLEG): container finished" podID="0a6f0aba-49c7-4dc2-8616-38aa09ead824" containerID="2460323dd6f3c2d88450279c4d390b76c7dfbf7f62bd7549e370733d6a1a0b77" exitCode=0
Nov 03 22:11:47.865318 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.865311    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"
Nov 03 22:11:47.865538 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.865336    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l" event={"ID":"0a6f0aba-49c7-4dc2-8616-38aa09ead824","Type":"ContainerDied","Data":"2460323dd6f3c2d88450279c4d390b76c7dfbf7f62bd7549e370733d6a1a0b77"}
Nov 03 22:11:47.865538 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.865396    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l" event={"ID":"0a6f0aba-49c7-4dc2-8616-38aa09ead824","Type":"ContainerDied","Data":"8d04ad341f2dec91cecbf12e61cf8d5ef23fa249dc3840fc131b8c5d28a64a1f"}
Nov 03 22:11:47.865538 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.865430    5053 scope.go:122] "RemoveContainer" containerID="2460323dd6f3c2d88450279c4d390b76c7dfbf7f62bd7549e370733d6a1a0b77"
Nov 03 22:11:47.876202 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.876179    5053 scope.go:122] "RemoveContainer" containerID="2460323dd6f3c2d88450279c4d390b76c7dfbf7f62bd7549e370733d6a1a0b77"
Nov 03 22:11:47.876581 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:11:47.876556    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2460323dd6f3c2d88450279c4d390b76c7dfbf7f62bd7549e370733d6a1a0b77\": container with ID starting with 2460323dd6f3c2d88450279c4d390b76c7dfbf7f62bd7549e370733d6a1a0b77 not found: ID does not exist" containerID="2460323dd6f3c2d88450279c4d390b76c7dfbf7f62bd7549e370733d6a1a0b77"
Nov 03 22:11:47.876642 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:47.876605    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2460323dd6f3c2d88450279c4d390b76c7dfbf7f62bd7549e370733d6a1a0b77"} err="failed to get container status \"2460323dd6f3c2d88450279c4d390b76c7dfbf7f62bd7549e370733d6a1a0b77\": rpc error: code = NotFound desc = could not find container \"2460323dd6f3c2d88450279c4d390b76c7dfbf7f62bd7549e370733d6a1a0b77\": container with ID starting with 2460323dd6f3c2d88450279c4d390b76c7dfbf7f62bd7549e370733d6a1a0b77 not found: ID does not exist"
Nov 03 22:11:48.030504 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:48.030467    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"]
Nov 03 22:11:48.033385 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:48.033359    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-79e7c7-openshift-default-6fc8dd6b49-dd24l"]
Nov 03 22:11:48.086679 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:48.086644    5053 scope.go:122] "RemoveContainer" containerID="adebc406e52a89ef1caf82455df721f175bb2e48cef6226d8aae178b3741a02c"
Nov 03 22:11:48.096843 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:48.096820    5053 scope.go:122] "RemoveContainer" containerID="306bf147c5f92c74cfd720cd41ab665a743ccfe77af9159989a0d38672e2c126"
Nov 03 22:11:49.668477 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:49.668441    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a6f0aba-49c7-4dc2-8616-38aa09ead824" path="/var/lib/kubelet/pods/0a6f0aba-49c7-4dc2-8616-38aa09ead824/volumes"
Nov 03 22:11:58.798163 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:58.798122    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"]
Nov 03 22:11:58.798860 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:58.798842    5053 cpu_manager.go:414] "containerMap: removing container" podUID="7ee6091e-7d0a-43b8-836e-9565f25ecc5b" containerName="istio-proxy"
Nov 03 22:11:58.798906 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:58.798864    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee6091e-7d0a-43b8-836e-9565f25ecc5b" containerName="istio-proxy"
Nov 03 22:11:58.798906 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:58.798892    5053 cpu_manager.go:414] "containerMap: removing container" podUID="0a6f0aba-49c7-4dc2-8616-38aa09ead824" containerName="istio-proxy"
Nov 03 22:11:58.798906 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:58.798897    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a6f0aba-49c7-4dc2-8616-38aa09ead824" containerName="istio-proxy"
Nov 03 22:11:58.799021 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:58.799010    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee6091e-7d0a-43b8-836e-9565f25ecc5b" containerName="istio-proxy"
Nov 03 22:11:58.799058 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:58.799028    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a6f0aba-49c7-4dc2-8616-38aa09ead824" containerName="istio-proxy"
Nov 03 22:11:58.804221 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:58.804200    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"
Nov 03 22:11:58.859833 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:58.859794    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"]
Nov 03 22:11:58.904298 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:58.904261    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-12dee4-serving-cert\""
Nov 03 22:11:58.979475 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:58.979442    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4vs9\" (UniqueName: \"kubernetes.io/projected/79ff394d-d937-46b1-b994-e180a924a3e0-kube-api-access-h4vs9\") pod \"maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm\" (UID: \"79ff394d-d937-46b1-b994-e180a924a3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"
Nov 03 22:11:58.979680 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:58.979652    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ff394d-d937-46b1-b994-e180a924a3e0-maas-api-tls\") pod \"maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm\" (UID: \"79ff394d-d937-46b1-b994-e180a924a3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"
Nov 03 22:11:58.979753 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:58.979688    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ff394d-d937-46b1-b994-e180a924a3e0-openshift-service-ca\") pod \"maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm\" (UID: \"79ff394d-d937-46b1-b994-e180a924a3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"
Nov 03 22:11:58.979877 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:58.979849    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ff394d-d937-46b1-b994-e180a924a3e0-metrics-certs\") pod \"maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm\" (UID: \"79ff394d-d937-46b1-b994-e180a924a3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"
Nov 03 22:11:59.081073 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:59.080981    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ff394d-d937-46b1-b994-e180a924a3e0-metrics-certs\") pod \"maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm\" (UID: \"79ff394d-d937-46b1-b994-e180a924a3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"
Nov 03 22:11:59.081305 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:59.081080    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4vs9\" (UniqueName: \"kubernetes.io/projected/79ff394d-d937-46b1-b994-e180a924a3e0-kube-api-access-h4vs9\") pod \"maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm\" (UID: \"79ff394d-d937-46b1-b994-e180a924a3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"
Nov 03 22:11:59.081305 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:59.081218    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ff394d-d937-46b1-b994-e180a924a3e0-maas-api-tls\") pod \"maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm\" (UID: \"79ff394d-d937-46b1-b994-e180a924a3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"
Nov 03 22:11:59.081418 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:59.081303    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ff394d-d937-46b1-b994-e180a924a3e0-openshift-service-ca\") pod \"maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm\" (UID: \"79ff394d-d937-46b1-b994-e180a924a3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"
Nov 03 22:11:59.081418 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:11:59.081367    5053 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-serving-cert: secret "maas-api-e2e-autores-12dee4-serving-cert" not found
Nov 03 22:11:59.081496 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:11:59.081456    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79ff394d-d937-46b1-b994-e180a924a3e0-maas-api-tls podName:79ff394d-d937-46b1-b994-e180a924a3e0 nodeName:}" failed. No retries permitted until 2026-11-03 22:11:59.581432901 +0000 UTC m=+3434.075354773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/79ff394d-d937-46b1-b994-e180a924a3e0-maas-api-tls") pod "maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm" (UID: "79ff394d-d937-46b1-b994-e180a924a3e0") : secret "maas-api-e2e-autores-12dee4-serving-cert" not found
Nov 03 22:11:59.081918 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:59.081899    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ff394d-d937-46b1-b994-e180a924a3e0-openshift-service-ca\") pod \"maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm\" (UID: \"79ff394d-d937-46b1-b994-e180a924a3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"
Nov 03 22:11:59.084469 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:59.084447    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ff394d-d937-46b1-b994-e180a924a3e0-metrics-certs\") pod \"maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm\" (UID: \"79ff394d-d937-46b1-b994-e180a924a3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"
Nov 03 22:11:59.138905 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:59.138872    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4vs9\" (UniqueName: \"kubernetes.io/projected/79ff394d-d937-46b1-b994-e180a924a3e0-kube-api-access-h4vs9\") pod \"maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm\" (UID: \"79ff394d-d937-46b1-b994-e180a924a3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"
Nov 03 22:11:59.588592 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:59.588555    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ff394d-d937-46b1-b994-e180a924a3e0-maas-api-tls\") pod \"maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm\" (UID: \"79ff394d-d937-46b1-b994-e180a924a3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"
Nov 03 22:11:59.591387 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:59.591352    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ff394d-d937-46b1-b994-e180a924a3e0-maas-api-tls\") pod \"maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm\" (UID: \"79ff394d-d937-46b1-b994-e180a924a3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"
Nov 03 22:11:59.719100 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:11:59.719055    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"
Nov 03 22:12:00.099058 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:00.099024    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"]
Nov 03 22:12:00.100420 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:12:00.100390    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ff394d_d937_46b1_b994_e180a924a3e0.slice/crio-443ff47084f2cdf1c4352d984bfc0ee39825c0d2b9faf9231847b07cb9a898c0 WatchSource:0}: Error finding container 443ff47084f2cdf1c4352d984bfc0ee39825c0d2b9faf9231847b07cb9a898c0: Status 404 returned error can't find the container with id 443ff47084f2cdf1c4352d984bfc0ee39825c0d2b9faf9231847b07cb9a898c0
Nov 03 22:12:00.102058 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:00.102039    5053 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 22:12:00.960905 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:00.960867    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm" event={"ID":"79ff394d-d937-46b1-b994-e180a924a3e0","Type":"ContainerStarted","Data":"443ff47084f2cdf1c4352d984bfc0ee39825c0d2b9faf9231847b07cb9a898c0"}
Nov 03 22:12:01.968722 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:01.968682    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm" event={"ID":"79ff394d-d937-46b1-b994-e180a924a3e0","Type":"ContainerStarted","Data":"81b7126a261fb1605a2c7cd2b5de5cb407bdfd34dfa06f0e90ce1f51298ee76e"}
Nov 03 22:12:01.969142 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:01.968760    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"
Nov 03 22:12:02.086611 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.086555    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm" podStartSLOduration=3.175352758 podStartE2EDuration="4.08654022s" totalImagesPullingTime="911.187462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:11:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:02.083936721 +0000 UTC m=+3436.577858597" watchObservedRunningTime="2026-11-03 22:12:02.08654022 +0000 UTC m=+3436.580462139"
Nov 03 22:12:02.639228 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.639180    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l" podUID="ad53efb5-e967-4315-8585-56ff322bb2b3" containerName="main" containerID="cri-o://5713dcaa702180b9d526dfc4892ed16dcf625c6750c1560b9bd9d4aabc584cea" gracePeriod=45
Nov 03 22:12:02.927885 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.927855    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:12:02.936130 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.936095    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad53efb5-e967-4315-8585-56ff322bb2b3-kube-api-access-qdlb5\" (UniqueName: \"kubernetes.io/projected/ad53efb5-e967-4315-8585-56ff322bb2b3-kube-api-access-qdlb5\") pod \"ad53efb5-e967-4315-8585-56ff322bb2b3\" (UID: \"ad53efb5-e967-4315-8585-56ff322bb2b3\") "
Nov 03 22:12:02.936310 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.936171    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-dshm\") pod \"ad53efb5-e967-4315-8585-56ff322bb2b3\" (UID: \"ad53efb5-e967-4315-8585-56ff322bb2b3\") "
Nov 03 22:12:02.936310 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.936282    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-tmp-dir\") pod \"ad53efb5-e967-4315-8585-56ff322bb2b3\" (UID: \"ad53efb5-e967-4315-8585-56ff322bb2b3\") "
Nov 03 22:12:02.936443 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.936427    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-home\" (UniqueName: \"kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-home\") pod \"ad53efb5-e967-4315-8585-56ff322bb2b3\" (UID: \"ad53efb5-e967-4315-8585-56ff322bb2b3\") "
Nov 03 22:12:02.936493 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.936462    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad53efb5-e967-4315-8585-56ff322bb2b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad53efb5-e967-4315-8585-56ff322bb2b3-tls-certs\") pod \"ad53efb5-e967-4315-8585-56ff322bb2b3\" (UID: \"ad53efb5-e967-4315-8585-56ff322bb2b3\") "
Nov 03 22:12:02.936558 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.936496    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-model-cache\") pod \"ad53efb5-e967-4315-8585-56ff322bb2b3\" (UID: \"ad53efb5-e967-4315-8585-56ff322bb2b3\") "
Nov 03 22:12:02.936618 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.936566    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-tmp-dir" pod "ad53efb5-e967-4315-8585-56ff322bb2b3" (UID: "ad53efb5-e967-4315-8585-56ff322bb2b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:12:02.936894 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.936707    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-home" pod "ad53efb5-e967-4315-8585-56ff322bb2b3" (UID: "ad53efb5-e967-4315-8585-56ff322bb2b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:12:02.936894 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.936867    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-model-cache" pod "ad53efb5-e967-4315-8585-56ff322bb2b3" (UID: "ad53efb5-e967-4315-8585-56ff322bb2b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:12:02.937021 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.936994    5053 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-home\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:12:02.937021 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.937014    5053 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-model-cache\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:12:02.937110 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.937028    5053 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-tmp-dir\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:12:02.939136 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.938929    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad53efb5-e967-4315-8585-56ff322bb2b3-tls-certs" pod "ad53efb5-e967-4315-8585-56ff322bb2b3" (UID: "ad53efb5-e967-4315-8585-56ff322bb2b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:12:02.939136 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.939086    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad53efb5-e967-4315-8585-56ff322bb2b3-kube-api-access-qdlb5" pod "ad53efb5-e967-4315-8585-56ff322bb2b3" (UID: "ad53efb5-e967-4315-8585-56ff322bb2b3"). InnerVolumeSpecName "kube-api-access-qdlb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:12:02.939625 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.939456    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-dshm" pod "ad53efb5-e967-4315-8585-56ff322bb2b3" (UID: "ad53efb5-e967-4315-8585-56ff322bb2b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:12:02.987013 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.986972    5053 generic.go:356] "Generic (PLEG): container finished" podID="ad53efb5-e967-4315-8585-56ff322bb2b3" containerID="5713dcaa702180b9d526dfc4892ed16dcf625c6750c1560b9bd9d4aabc584cea" exitCode=0
Nov 03 22:12:02.987521 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.987038    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"
Nov 03 22:12:02.987521 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.987065    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l" event={"ID":"ad53efb5-e967-4315-8585-56ff322bb2b3","Type":"ContainerDied","Data":"5713dcaa702180b9d526dfc4892ed16dcf625c6750c1560b9bd9d4aabc584cea"}
Nov 03 22:12:02.987521 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.987113    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l" event={"ID":"ad53efb5-e967-4315-8585-56ff322bb2b3","Type":"ContainerDied","Data":"f8a5a800b153f7e3caf83263acf84ea08d6653737b6068015b9fc11eceed1942"}
Nov 03 22:12:02.987521 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.987132    5053 scope.go:122] "RemoveContainer" containerID="5713dcaa702180b9d526dfc4892ed16dcf625c6750c1560b9bd9d4aabc584cea"
Nov 03 22:12:02.998656 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.998633    5053 scope.go:122] "RemoveContainer" containerID="5713dcaa702180b9d526dfc4892ed16dcf625c6750c1560b9bd9d4aabc584cea"
Nov 03 22:12:02.999011 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:12:02.998988    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5713dcaa702180b9d526dfc4892ed16dcf625c6750c1560b9bd9d4aabc584cea\": container with ID starting with 5713dcaa702180b9d526dfc4892ed16dcf625c6750c1560b9bd9d4aabc584cea not found: ID does not exist" containerID="5713dcaa702180b9d526dfc4892ed16dcf625c6750c1560b9bd9d4aabc584cea"
Nov 03 22:12:02.999073 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:02.999021    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5713dcaa702180b9d526dfc4892ed16dcf625c6750c1560b9bd9d4aabc584cea"} err="failed to get container status \"5713dcaa702180b9d526dfc4892ed16dcf625c6750c1560b9bd9d4aabc584cea\": rpc error: code = NotFound desc = could not find container \"5713dcaa702180b9d526dfc4892ed16dcf625c6750c1560b9bd9d4aabc584cea\": container with ID starting with 5713dcaa702180b9d526dfc4892ed16dcf625c6750c1560b9bd9d4aabc584cea not found: ID does not exist"
Nov 03 22:12:03.037730 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.037695    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdlb5\" (UniqueName: \"kubernetes.io/projected/ad53efb5-e967-4315-8585-56ff322bb2b3-kube-api-access-qdlb5\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:12:03.037948 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.037929    5053 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad53efb5-e967-4315-8585-56ff322bb2b3-dshm\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:12:03.038014 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.037953    5053 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad53efb5-e967-4315-8585-56ff322bb2b3-tls-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:12:03.153840 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.153723    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"]
Nov 03 22:12:03.159130 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.159071    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-79e7c7/noten-model-79e7c7-kserve-b48b6dbb9-f9w9l"]
Nov 03 22:12:03.668726 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.668691    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad53efb5-e967-4315-8585-56ff322bb2b3" path="/var/lib/kubelet/pods/ad53efb5-e967-4315-8585-56ff322bb2b3/volumes"
Nov 03 22:12:03.726854 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.726812    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"]
Nov 03 22:12:03.727802 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.727775    5053 cpu_manager.go:414] "containerMap: removing container" podUID="ad53efb5-e967-4315-8585-56ff322bb2b3" containerName="main"
Nov 03 22:12:03.727802 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.727799    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad53efb5-e967-4315-8585-56ff322bb2b3" containerName="main"
Nov 03 22:12:03.728019 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.728005    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad53efb5-e967-4315-8585-56ff322bb2b3" containerName="main"
Nov 03 22:12:03.733480 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.733455    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:12:03.745775 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.745744    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-tls-certs\") pod \"auth-test-model-7fb477-kserve-57d9d7b457-jhl48\" (UID: \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\") " pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:12:03.745971 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.745844    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-tmp-dir\") pod \"auth-test-model-7fb477-kserve-57d9d7b457-jhl48\" (UID: \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\") " pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:12:03.746060 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.745965    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tplnx\" (UniqueName: \"kubernetes.io/projected/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-kube-api-access-tplnx\") pod \"auth-test-model-7fb477-kserve-57d9d7b457-jhl48\" (UID: \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\") " pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:12:03.746060 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.746043    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-dshm\") pod \"auth-test-model-7fb477-kserve-57d9d7b457-jhl48\" (UID: \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\") " pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:12:03.746188 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.746078    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-model-cache\") pod \"auth-test-model-7fb477-kserve-57d9d7b457-jhl48\" (UID: \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\") " pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:12:03.746188 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.746165    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-home\") pod \"auth-test-model-7fb477-kserve-57d9d7b457-jhl48\" (UID: \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\") " pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:12:03.786007 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.785353    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7fb477\"/\"kube-root-ca.crt\""
Nov 03 22:12:03.786007 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.785413    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7fb477\"/\"openshift-service-ca.crt\""
Nov 03 22:12:03.786007 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.785906    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"]
Nov 03 22:12:03.830229 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.830193    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7fb477\"/\"auth-test-model-7fb477-kserve-self-signed-certs\""
Nov 03 22:12:03.830463 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.830228    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7fb477\"/\"default-dockercfg-k7tvq\""
Nov 03 22:12:03.847853 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.847817    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-tls-certs\") pod \"auth-test-model-7fb477-kserve-57d9d7b457-jhl48\" (UID: \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\") " pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:12:03.848076 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.847966    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-tmp-dir\") pod \"auth-test-model-7fb477-kserve-57d9d7b457-jhl48\" (UID: \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\") " pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:12:03.848076 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.848000    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tplnx\" (UniqueName: \"kubernetes.io/projected/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-kube-api-access-tplnx\") pod \"auth-test-model-7fb477-kserve-57d9d7b457-jhl48\" (UID: \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\") " pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:12:03.848076 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.848037    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-dshm\") pod \"auth-test-model-7fb477-kserve-57d9d7b457-jhl48\" (UID: \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\") " pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:12:03.848402 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.848082    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-model-cache\") pod \"auth-test-model-7fb477-kserve-57d9d7b457-jhl48\" (UID: \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\") " pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:12:03.848402 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.848139    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-home\") pod \"auth-test-model-7fb477-kserve-57d9d7b457-jhl48\" (UID: \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\") " pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:12:03.849065 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.849039    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-tmp-dir\") pod \"auth-test-model-7fb477-kserve-57d9d7b457-jhl48\" (UID: \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\") " pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:12:03.849275 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.849256    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-model-cache\") pod \"auth-test-model-7fb477-kserve-57d9d7b457-jhl48\" (UID: \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\") " pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:12:03.850509 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.850476    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-home\") pod \"auth-test-model-7fb477-kserve-57d9d7b457-jhl48\" (UID: \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\") " pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:12:03.851439 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.851411    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-dshm\") pod \"auth-test-model-7fb477-kserve-57d9d7b457-jhl48\" (UID: \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\") " pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:12:03.851573 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.851559    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-tls-certs\") pod \"auth-test-model-7fb477-kserve-57d9d7b457-jhl48\" (UID: \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\") " pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:12:03.909722 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:03.909685    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tplnx\" (UniqueName: \"kubernetes.io/projected/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-kube-api-access-tplnx\") pod \"auth-test-model-7fb477-kserve-57d9d7b457-jhl48\" (UID: \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\") " pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:12:04.047512 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:04.047477    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:12:04.438949 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:04.438916    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"]
Nov 03 22:12:04.440126 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:12:04.440090    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9b7982f_9598_4fb0_b1d6_10a59db8b02b.slice/crio-4b489f671701c6fe3c3213299caedc3a51f9fd9def1aae561d7db0557b82b6df WatchSource:0}: Error finding container 4b489f671701c6fe3c3213299caedc3a51f9fd9def1aae561d7db0557b82b6df: Status 404 returned error can't find the container with id 4b489f671701c6fe3c3213299caedc3a51f9fd9def1aae561d7db0557b82b6df
Nov 03 22:12:05.004732 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:05.004696    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48" event={"ID":"e9b7982f-9598-4fb0-b1d6-10a59db8b02b","Type":"ContainerStarted","Data":"f6dced47f640e3c6536b9ceb298285ec2419515f209a3b4790e05147348583de"}
Nov 03 22:12:05.004732 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:05.004741    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48" event={"ID":"e9b7982f-9598-4fb0-b1d6-10a59db8b02b","Type":"ContainerStarted","Data":"4b489f671701c6fe3c3213299caedc3a51f9fd9def1aae561d7db0557b82b6df"}
Nov 03 22:12:05.048059 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:05.048020    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:12:05.122721 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:05.122669    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48" podStartSLOduration=2.122651419 podStartE2EDuration="2.122651419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:05.118523601 +0000 UTC m=+3439.612445481" watchObservedRunningTime="2026-11-03 22:12:05.122651419 +0000 UTC m=+3439.616573297"
Nov 03 22:12:07.993774 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:07.993745    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"
Nov 03 22:12:10.930249 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:12:10.930191    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:12:14.048604 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:14.048562    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:12:14.062877 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:14.062847    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:12:14.079975 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:14.079935    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:12:28.957534 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:28.957433    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"]
Nov 03 22:12:28.962064 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:28.962040    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:28.975982 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:28.975932    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-envoy\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:28.976163 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:28.975997    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-podinfo\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:28.976163 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:28.976145    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-data\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:28.976303 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:28.976190    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istiod-ca-cert\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:28.976303 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:28.976286    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-token\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:28.976398 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:28.976309    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-workload-socket\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:28.976398 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:28.976347    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-credential-socket\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:28.976484 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:28.976451    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvnpr\" (UniqueName: \"kubernetes.io/projected/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-kube-api-access-qvnpr\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:28.976535 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:28.976493    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-workload-certs\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:29.045314 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.045276    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-5b1489-openshift-default-dockercfg-74kvx\""
Nov 03 22:12:29.068069 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.068038    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"]
Nov 03 22:12:29.077154 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.077113    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-credential-socket\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:29.077381 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.077172    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvnpr\" (UniqueName: \"kubernetes.io/projected/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-kube-api-access-qvnpr\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:29.077381 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.077225    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-workload-certs\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:29.077381 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.077371    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-envoy\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:29.077578 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.077414    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-podinfo\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:29.077578 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.077508    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-data\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:29.077578 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.077540    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istiod-ca-cert\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:29.077578 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.077569    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-token\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:29.077795 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.077594    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-workload-socket\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:29.077795 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.077703    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-credential-socket\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:29.077985 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.077963    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-workload-socket\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:29.078971 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.078945    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-workload-certs\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:29.080096 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.079922    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istiod-ca-cert\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:29.080293 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.080041    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-data\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:29.081476 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.081454    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-envoy\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:29.084890 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.084865    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-podinfo\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:29.169977 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.169907    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-token\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:29.213648 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.213569    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvnpr\" (UniqueName: \"kubernetes.io/projected/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-kube-api-access-qvnpr\") pod \"e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") " pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:29.277591 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.277558    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:29.731141 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:12:29.731107    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f17d174_9d7f_4b54_82c5_6c4dac27b48b.slice/crio-48d799c7785a7f70ab4469d4363b6b828824bc65e5f02bd3a7a994549634dec3 WatchSource:0}: Error finding container 48d799c7785a7f70ab4469d4363b6b828824bc65e5f02bd3a7a994549634dec3: Status 404 returned error can't find the container with id 48d799c7785a7f70ab4469d4363b6b828824bc65e5f02bd3a7a994549634dec3
Nov 03 22:12:29.731320 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.731297    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"]
Nov 03 22:12:29.734118 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.734079    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:12:29.734258 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.734153    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:12:29.734258 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:29.734188    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:12:30.201598 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:30.201556    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk" event={"ID":"4f17d174-9d7f-4b54-82c5-6c4dac27b48b","Type":"ContainerStarted","Data":"1ee2cd4dca86f76a4e71731677982d4a651f35d518a258a82243b0e1ce32b22e"}
Nov 03 22:12:30.202100 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:30.201625    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk" event={"ID":"4f17d174-9d7f-4b54-82c5-6c4dac27b48b","Type":"ContainerStarted","Data":"48d799c7785a7f70ab4469d4363b6b828824bc65e5f02bd3a7a994549634dec3"}
Nov 03 22:12:30.278006 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:30.277967    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:30.284469 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:30.284437    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:30.318454 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:30.318393    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk" podStartSLOduration=2.318376357 podStartE2EDuration="2.318376357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:30.317679248 +0000 UTC m=+3464.811601167" watchObservedRunningTime="2026-11-03 22:12:30.318376357 +0000 UTC m=+3464.812298232"
Nov 03 22:12:31.209126 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:31.209078    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:31.210310 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:31.210290    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:12:40.931018 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:12:40.930975    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:12:54.468481 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:54.468430    5053 patch_prober.go:29] interesting pod/e2e-autores-12dee4-openshift-default-577996447c-jqml4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:12:54.469037 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:54.468509    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4" podUID="6862760e-4710-4dc1-afd1-30df93440970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:12:56.453594 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:56.453556    5053 generic.go:356] "Generic (PLEG): container finished" podID="6862760e-4710-4dc1-afd1-30df93440970" containerID="7e251468046f1aa4bec1d2389207a5003941723afbc5f1f63f051afecf56d636" exitCode=0
Nov 03 22:12:56.454072 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:56.453628    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4" event={"ID":"6862760e-4710-4dc1-afd1-30df93440970","Type":"ContainerDied","Data":"7e251468046f1aa4bec1d2389207a5003941723afbc5f1f63f051afecf56d636"}
Nov 03 22:12:56.454120 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:56.454100    5053 scope.go:122] "RemoveContainer" containerID="7e251468046f1aa4bec1d2389207a5003941723afbc5f1f63f051afecf56d636"
Nov 03 22:12:56.456015 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:56.455989    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:12:56.456086 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:56.456058    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:12:56.456129 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:56.456091    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:12:57.471160 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:57.471122    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4" event={"ID":"6862760e-4710-4dc1-afd1-30df93440970","Type":"ContainerStarted","Data":"3d0794c985e0d28a7bdeed4f07be41eeccdd896e93007e7f77deede303965ade"}
Nov 03 22:12:57.554158 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:57.554109    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:12:57.554404 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:57.554360    5053 patch_prober.go:29] interesting pod/e2e-autores-12dee4-openshift-default-577996447c-jqml4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Nov 03 22:12:57.554476 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:57.554447    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4" podUID="6862760e-4710-4dc1-afd1-30df93440970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Nov 03 22:12:58.554749 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:58.554713    5053 patch_prober.go:29] interesting pod/e2e-autores-12dee4-openshift-default-577996447c-jqml4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Nov 03 22:12:58.555281 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:58.554777    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4" podUID="6862760e-4710-4dc1-afd1-30df93440970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Nov 03 22:12:59.554830 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:59.554796    5053 patch_prober.go:29] interesting pod/e2e-autores-12dee4-openshift-default-577996447c-jqml4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Nov 03 22:12:59.555274 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:12:59.554874    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4" podUID="6862760e-4710-4dc1-afd1-30df93440970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Nov 03 22:13:00.554277 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:00.554212    5053 patch_prober.go:29] interesting pod/e2e-autores-12dee4-openshift-default-577996447c-jqml4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Nov 03 22:13:00.554539 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:00.554304    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4" podUID="6862760e-4710-4dc1-afd1-30df93440970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Nov 03 22:13:01.554396 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:01.554353    5053 patch_prober.go:29] interesting pod/e2e-autores-12dee4-openshift-default-577996447c-jqml4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Nov 03 22:13:01.554923 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:01.554431    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4" podUID="6862760e-4710-4dc1-afd1-30df93440970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Nov 03 22:13:02.554287 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:02.554227    5053 patch_prober.go:29] interesting pod/e2e-autores-12dee4-openshift-default-577996447c-jqml4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Nov 03 22:13:02.554489 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:02.554329    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4" podUID="6862760e-4710-4dc1-afd1-30df93440970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Nov 03 22:13:03.558519 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:03.558486    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:13:03.559023 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:03.558936    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:13:03.559685 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:03.559664    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:13:10.936064 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:13:10.936017    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:13:40.930470 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:13:40.930424    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:13:44.282032 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:44.282001    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx"]
Nov 03 22:13:44.286665 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:44.286644    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx"
Nov 03 22:13:44.338569 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:44.338530    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-5b1489-serving-cert\""
Nov 03 22:13:44.393406 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:44.393366    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx"]
Nov 03 22:13:44.449816 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:44.449777    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bw8z\" (UniqueName: \"kubernetes.io/projected/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-kube-api-access-9bw8z\") pod \"maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx\" (UID: \"a6d529fb-027e-4ce2-af6c-50a7f716ddb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx"
Nov 03 22:13:44.449991 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:44.449904    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx\" (UID: \"a6d529fb-027e-4ce2-af6c-50a7f716ddb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx"
Nov 03 22:13:44.449991 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:44.449978    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx\" (UID: \"a6d529fb-027e-4ce2-af6c-50a7f716ddb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx"
Nov 03 22:13:44.450131 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:44.450116    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx\" (UID: \"a6d529fb-027e-4ce2-af6c-50a7f716ddb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx"
Nov 03 22:13:44.551888 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:44.551808    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bw8z\" (UniqueName: \"kubernetes.io/projected/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-kube-api-access-9bw8z\") pod \"maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx\" (UID: \"a6d529fb-027e-4ce2-af6c-50a7f716ddb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx"
Nov 03 22:13:44.551888 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:44.551868    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx\" (UID: \"a6d529fb-027e-4ce2-af6c-50a7f716ddb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx"
Nov 03 22:13:44.552126 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:44.551932    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx\" (UID: \"a6d529fb-027e-4ce2-af6c-50a7f716ddb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx"
Nov 03 22:13:44.552126 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:44.552023    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx\" (UID: \"a6d529fb-027e-4ce2-af6c-50a7f716ddb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx"
Nov 03 22:13:44.552672 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:44.552640    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx\" (UID: \"a6d529fb-027e-4ce2-af6c-50a7f716ddb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx"
Nov 03 22:13:44.554780 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:44.554757    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx\" (UID: \"a6d529fb-027e-4ce2-af6c-50a7f716ddb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx"
Nov 03 22:13:44.554780 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:44.554775    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx\" (UID: \"a6d529fb-027e-4ce2-af6c-50a7f716ddb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx"
Nov 03 22:13:44.609882 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:44.609842    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bw8z\" (UniqueName: \"kubernetes.io/projected/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-kube-api-access-9bw8z\") pod \"maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx\" (UID: \"a6d529fb-027e-4ce2-af6c-50a7f716ddb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx"
Nov 03 22:13:44.902264 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:44.902146    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx"
Nov 03 22:13:45.291283 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:45.291250    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx"]
Nov 03 22:13:45.844599 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:45.844560    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx" event={"ID":"a6d529fb-027e-4ce2-af6c-50a7f716ddb9","Type":"ContainerStarted","Data":"89fbb74cffe802722e8e20b301026dccf862fde3c2a319aa4e0912db7fab2836"}
Nov 03 22:13:46.852383 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:46.852349    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx" event={"ID":"a6d529fb-027e-4ce2-af6c-50a7f716ddb9","Type":"ContainerStarted","Data":"5c16d72246dda09fb29d9702dea7e75e783339abd1d2dd915745e0a75664c847"}
Nov 03 22:13:46.852796 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:46.852428    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx"
Nov 03 22:13:46.970894 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:46.970829    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx" podStartSLOduration=2.003900995 podStartE2EDuration="2.970808473s" totalImagesPullingTime="966.907478ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:46.9665691 +0000 UTC m=+3541.460490976" watchObservedRunningTime="2026-11-03 22:13:46.970808473 +0000 UTC m=+3541.464730353"
Nov 03 22:13:52.865888 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:13:52.865855    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx"
Nov 03 22:14:10.931376 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:14:10.931338    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:14:30.183696 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:30.183612    5053 generic.go:356] "Generic (PLEG): container finished" podID="4f17d174-9d7f-4b54-82c5-6c4dac27b48b" containerID="1ee2cd4dca86f76a4e71731677982d4a651f35d518a258a82243b0e1ce32b22e" exitCode=0
Nov 03 22:14:30.184135 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:30.183686    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk" event={"ID":"4f17d174-9d7f-4b54-82c5-6c4dac27b48b","Type":"ContainerDied","Data":"1ee2cd4dca86f76a4e71731677982d4a651f35d518a258a82243b0e1ce32b22e"}
Nov 03 22:14:30.184135 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:30.184114    5053 scope.go:122] "RemoveContainer" containerID="1ee2cd4dca86f76a4e71731677982d4a651f35d518a258a82243b0e1ce32b22e"
Nov 03 22:14:30.185815 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:30.185786    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:14:30.185888 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:30.185870    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:14:30.185926 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:30.185904    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:14:31.191987 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:31.191952    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk" event={"ID":"4f17d174-9d7f-4b54-82c5-6c4dac27b48b","Type":"ContainerStarted","Data":"58af5eaa3e828c811720dc448ef8afe679b1da811eb26441d18cb8f10d16cf1f"}
Nov 03 22:14:31.211390 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:31.211348    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:14:31.278003 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:31.277973    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:14:31.279421 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:31.279394    5053 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Nov 03 22:14:31.279554 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:31.279456    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk" podUID="4f17d174-9d7f-4b54-82c5-6c4dac27b48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Nov 03 22:14:32.277982 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:32.277939    5053 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Nov 03 22:14:32.278490 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:32.277994    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk" podUID="4f17d174-9d7f-4b54-82c5-6c4dac27b48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Nov 03 22:14:33.278779 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:33.278743    5053 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Nov 03 22:14:33.279302 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:33.278834    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk" podUID="4f17d174-9d7f-4b54-82c5-6c4dac27b48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Nov 03 22:14:34.278541 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:34.278506    5053 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Nov 03 22:14:34.278740 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:34.278568    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk" podUID="4f17d174-9d7f-4b54-82c5-6c4dac27b48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Nov 03 22:14:35.278168 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:35.278133    5053 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Nov 03 22:14:35.278714 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:35.278190    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk" podUID="4f17d174-9d7f-4b54-82c5-6c4dac27b48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Nov 03 22:14:36.279007 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:36.278976    5053 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Nov 03 22:14:36.279526 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:36.279030    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk" podUID="4f17d174-9d7f-4b54-82c5-6c4dac27b48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Nov 03 22:14:37.282877 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:37.282846    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:14:37.284078 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:37.284058    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:14:40.928839 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:14:40.928802    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:14:46.284477 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:46.284449    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 22:14:46.289030 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:46.289006    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 22:14:46.293533 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:46.293509    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 22:14:46.297646 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:14:46.297624    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 22:15:00.184373 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.184338    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj"]
Nov 03 22:15:00.188934 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.188909    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw"]
Nov 03 22:15:00.189102 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.189020    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj"
Nov 03 22:15:00.192820 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.192798    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw"
Nov 03 22:15:00.221768 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.221394    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms"]
Nov 03 22:15:00.245519 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.245487    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt"]
Nov 03 22:15:00.252209 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.252179    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt"
Nov 03 22:15:00.252734 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.245598    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms"
Nov 03 22:15:00.263591 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.263561    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj"]
Nov 03 22:15:00.275553 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.275194    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw"]
Nov 03 22:15:00.283162 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.283138    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms"]
Nov 03 22:15:00.305340 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.305288    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f8d07a-8d67-4366-97b2-2773315a8ec0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895735-jd7qj\" (UID: \"94f8d07a-8d67-4366-97b2-2773315a8ec0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj"
Nov 03 22:15:00.305773 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.305744    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zsqd\" (UniqueName: \"kubernetes.io/projected/94f8d07a-8d67-4366-97b2-2773315a8ec0-kube-api-access-8zsqd\") pod \"maas-api-key-cleanup-29895735-jd7qj\" (UID: \"94f8d07a-8d67-4366-97b2-2773315a8ec0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj"
Nov 03 22:15:00.305898 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.305786    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5d77a67-7f9e-40ca-b67f-49b1be17239f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms\" (UID: \"b5d77a67-7f9e-40ca-b67f-49b1be17239f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms"
Nov 03 22:15:00.305898 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.305836    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6wqp\" (UniqueName: \"kubernetes.io/projected/b5d77a67-7f9e-40ca-b67f-49b1be17239f-kube-api-access-l6wqp\") pod \"maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms\" (UID: \"b5d77a67-7f9e-40ca-b67f-49b1be17239f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms"
Nov 03 22:15:00.306353 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.306313    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r9xx\" (UniqueName: \"kubernetes.io/projected/a83ccbc3-6590-43cd-bda1-1e8f478060c1-kube-api-access-7r9xx\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw\" (UID: \"a83ccbc3-6590-43cd-bda1-1e8f478060c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw"
Nov 03 22:15:00.306543 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.306516    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83ccbc3-6590-43cd-bda1-1e8f478060c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw\" (UID: \"a83ccbc3-6590-43cd-bda1-1e8f478060c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw"
Nov 03 22:15:00.316564 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.316499    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt"]
Nov 03 22:15:00.324129 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.323531    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:15:00.407888 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.407835    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f8d07a-8d67-4366-97b2-2773315a8ec0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895735-jd7qj\" (UID: \"94f8d07a-8d67-4366-97b2-2773315a8ec0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj"
Nov 03 22:15:00.408099 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.407906    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvrw6\" (UniqueName: \"kubernetes.io/projected/f5db8aaf-b362-4a27-b828-54655dff7ad1-kube-api-access-nvrw6\") pod \"maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt\" (UID: \"f5db8aaf-b362-4a27-b828-54655dff7ad1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt"
Nov 03 22:15:00.408188 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.408127    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zsqd\" (UniqueName: \"kubernetes.io/projected/94f8d07a-8d67-4366-97b2-2773315a8ec0-kube-api-access-8zsqd\") pod \"maas-api-key-cleanup-29895735-jd7qj\" (UID: \"94f8d07a-8d67-4366-97b2-2773315a8ec0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj"
Nov 03 22:15:00.408188 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.408172    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5d77a67-7f9e-40ca-b67f-49b1be17239f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms\" (UID: \"b5d77a67-7f9e-40ca-b67f-49b1be17239f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms"
Nov 03 22:15:00.408329 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.408225    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6wqp\" (UniqueName: \"kubernetes.io/projected/b5d77a67-7f9e-40ca-b67f-49b1be17239f-kube-api-access-l6wqp\") pod \"maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms\" (UID: \"b5d77a67-7f9e-40ca-b67f-49b1be17239f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms"
Nov 03 22:15:00.408329 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.408296    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r9xx\" (UniqueName: \"kubernetes.io/projected/a83ccbc3-6590-43cd-bda1-1e8f478060c1-kube-api-access-7r9xx\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw\" (UID: \"a83ccbc3-6590-43cd-bda1-1e8f478060c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw"
Nov 03 22:15:00.408441 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.408337    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83ccbc3-6590-43cd-bda1-1e8f478060c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw\" (UID: \"a83ccbc3-6590-43cd-bda1-1e8f478060c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw"
Nov 03 22:15:00.408441 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.408377    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5db8aaf-b362-4a27-b828-54655dff7ad1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt\" (UID: \"f5db8aaf-b362-4a27-b828-54655dff7ad1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt"
Nov 03 22:15:00.408651 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.408578    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f8d07a-8d67-4366-97b2-2773315a8ec0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895735-jd7qj\" (UID: \"94f8d07a-8d67-4366-97b2-2773315a8ec0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj"
Nov 03 22:15:00.409482 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.409461    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5d77a67-7f9e-40ca-b67f-49b1be17239f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms\" (UID: \"b5d77a67-7f9e-40ca-b67f-49b1be17239f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms"
Nov 03 22:15:00.409796 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.409777    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83ccbc3-6590-43cd-bda1-1e8f478060c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw\" (UID: \"a83ccbc3-6590-43cd-bda1-1e8f478060c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw"
Nov 03 22:15:00.465789 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.465758    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6wqp\" (UniqueName: \"kubernetes.io/projected/b5d77a67-7f9e-40ca-b67f-49b1be17239f-kube-api-access-l6wqp\") pod \"maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms\" (UID: \"b5d77a67-7f9e-40ca-b67f-49b1be17239f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms"
Nov 03 22:15:00.465966 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.465793    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r9xx\" (UniqueName: \"kubernetes.io/projected/a83ccbc3-6590-43cd-bda1-1e8f478060c1-kube-api-access-7r9xx\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw\" (UID: \"a83ccbc3-6590-43cd-bda1-1e8f478060c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw"
Nov 03 22:15:00.465966 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.465767    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zsqd\" (UniqueName: \"kubernetes.io/projected/94f8d07a-8d67-4366-97b2-2773315a8ec0-kube-api-access-8zsqd\") pod \"maas-api-key-cleanup-29895735-jd7qj\" (UID: \"94f8d07a-8d67-4366-97b2-2773315a8ec0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj"
Nov 03 22:15:00.506035 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.506003    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj"
Nov 03 22:15:00.510563 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.510144    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvrw6\" (UniqueName: \"kubernetes.io/projected/f5db8aaf-b362-4a27-b828-54655dff7ad1-kube-api-access-nvrw6\") pod \"maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt\" (UID: \"f5db8aaf-b362-4a27-b828-54655dff7ad1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt"
Nov 03 22:15:00.510563 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.510346    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5db8aaf-b362-4a27-b828-54655dff7ad1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt\" (UID: \"f5db8aaf-b362-4a27-b828-54655dff7ad1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt"
Nov 03 22:15:00.511159 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.511136    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5db8aaf-b362-4a27-b828-54655dff7ad1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt\" (UID: \"f5db8aaf-b362-4a27-b828-54655dff7ad1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt"
Nov 03 22:15:00.524870 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.524839    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw"
Nov 03 22:15:00.567538 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.567480    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvrw6\" (UniqueName: \"kubernetes.io/projected/f5db8aaf-b362-4a27-b828-54655dff7ad1-kube-api-access-nvrw6\") pod \"maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt\" (UID: \"f5db8aaf-b362-4a27-b828-54655dff7ad1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt"
Nov 03 22:15:00.592324 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.592225    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt"
Nov 03 22:15:00.609578 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.609352    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms"
Nov 03 22:15:00.894409 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.894375    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj"]
Nov 03 22:15:00.895331 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:15:00.895306    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f8d07a_8d67_4366_97b2_2773315a8ec0.slice/crio-b9152a4f4d020d59b49c1f4e946448eabfac0ed7cbaf989bf1d90c245005f8b9 WatchSource:0}: Error finding container b9152a4f4d020d59b49c1f4e946448eabfac0ed7cbaf989bf1d90c245005f8b9: Status 404 returned error can't find the container with id b9152a4f4d020d59b49c1f4e946448eabfac0ed7cbaf989bf1d90c245005f8b9
Nov 03 22:15:00.949276 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:15:00.949209    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83ccbc3_6590_43cd_bda1_1e8f478060c1.slice/crio-728b1ce7e5b4146888f97be26db76f2ef83bf14644e50190d2af425c43025625 WatchSource:0}: Error finding container 728b1ce7e5b4146888f97be26db76f2ef83bf14644e50190d2af425c43025625: Status 404 returned error can't find the container with id 728b1ce7e5b4146888f97be26db76f2ef83bf14644e50190d2af425c43025625
Nov 03 22:15:00.950026 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:00.950001    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw"]
Nov 03 22:15:01.024853 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:01.024822    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt"]
Nov 03 22:15:01.025591 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:15:01.025563    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5db8aaf_b362_4a27_b828_54655dff7ad1.slice/crio-fd8f35c6c6347a280283193cfa956e07b0b712f81836f12e3f778181828934e9 WatchSource:0}: Error finding container fd8f35c6c6347a280283193cfa956e07b0b712f81836f12e3f778181828934e9: Status 404 returned error can't find the container with id fd8f35c6c6347a280283193cfa956e07b0b712f81836f12e3f778181828934e9
Nov 03 22:15:01.054065 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:01.054029    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms"]
Nov 03 22:15:01.068742 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:15:01.068714    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d77a67_7f9e_40ca_b67f_49b1be17239f.slice/crio-bf4f421d46d581907763d1940283ba465b4381a5904554e785f4a11947e440f1 WatchSource:0}: Error finding container bf4f421d46d581907763d1940283ba465b4381a5904554e785f4a11947e440f1: Status 404 returned error can't find the container with id bf4f421d46d581907763d1940283ba465b4381a5904554e785f4a11947e440f1
Nov 03 22:15:01.494108 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:01.494014    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt" event={"ID":"f5db8aaf-b362-4a27-b828-54655dff7ad1","Type":"ContainerStarted","Data":"fd8f35c6c6347a280283193cfa956e07b0b712f81836f12e3f778181828934e9"}
Nov 03 22:15:01.498215 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:01.498112    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj" event={"ID":"94f8d07a-8d67-4366-97b2-2773315a8ec0","Type":"ContainerStarted","Data":"5a471f74d21ea32dcabd7ee871d60d4c117a89430e258f313fc38f8b1e9d7beb"}
Nov 03 22:15:01.498215 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:01.498182    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj" event={"ID":"94f8d07a-8d67-4366-97b2-2773315a8ec0","Type":"ContainerStarted","Data":"b9152a4f4d020d59b49c1f4e946448eabfac0ed7cbaf989bf1d90c245005f8b9"}
Nov 03 22:15:01.499883 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:01.499837    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms" event={"ID":"b5d77a67-7f9e-40ca-b67f-49b1be17239f","Type":"ContainerStarted","Data":"bf4f421d46d581907763d1940283ba465b4381a5904554e785f4a11947e440f1"}
Nov 03 22:15:01.504062 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:01.502626    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw" event={"ID":"a83ccbc3-6590-43cd-bda1-1e8f478060c1","Type":"ContainerStarted","Data":"bd8cdc7b82a4f0a5f59f28a2931d1534cff74210c9cc2e92a7bc3fa32c510116"}
Nov 03 22:15:01.504062 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:01.502675    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw" event={"ID":"a83ccbc3-6590-43cd-bda1-1e8f478060c1","Type":"ContainerStarted","Data":"728b1ce7e5b4146888f97be26db76f2ef83bf14644e50190d2af425c43025625"}
Nov 03 22:15:01.735123 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:01.735058    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj" podStartSLOduration=1.735038276 podStartE2EDuration="1.735038276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:01.619541303 +0000 UTC m=+3616.113463180" watchObservedRunningTime="2026-11-03 22:15:01.735038276 +0000 UTC m=+3616.228960153"
Nov 03 22:15:01.737425 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:01.736620    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw" podStartSLOduration=1.73660692 podStartE2EDuration="1.73660692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:01.735037896 +0000 UTC m=+3616.228959772" watchObservedRunningTime="2026-11-03 22:15:01.73660692 +0000 UTC m=+3616.230528799"
Nov 03 22:15:02.513645 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:02.513608    5053 generic.go:356] "Generic (PLEG): container finished" podID="f5db8aaf-b362-4a27-b828-54655dff7ad1" containerID="5ff7fabf7740c8b1b6d73108f0e913adbabb7bb33a4f4c091ae73f41b93e1156" exitCode=60
Nov 03 22:15:02.514148 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:02.513704    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt" event={"ID":"f5db8aaf-b362-4a27-b828-54655dff7ad1","Type":"ContainerDied","Data":"5ff7fabf7740c8b1b6d73108f0e913adbabb7bb33a4f4c091ae73f41b93e1156"}
Nov 03 22:15:02.514148 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:02.514016    5053 scope.go:122] "RemoveContainer" containerID="5ff7fabf7740c8b1b6d73108f0e913adbabb7bb33a4f4c091ae73f41b93e1156"
Nov 03 22:15:02.515728 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:02.515697    5053 generic.go:356] "Generic (PLEG): container finished" podID="94f8d07a-8d67-4366-97b2-2773315a8ec0" containerID="5a471f74d21ea32dcabd7ee871d60d4c117a89430e258f313fc38f8b1e9d7beb" exitCode=60
Nov 03 22:15:02.515867 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:02.515780    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj" event={"ID":"94f8d07a-8d67-4366-97b2-2773315a8ec0","Type":"ContainerDied","Data":"5a471f74d21ea32dcabd7ee871d60d4c117a89430e258f313fc38f8b1e9d7beb"}
Nov 03 22:15:02.516078 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:02.516030    5053 scope.go:122] "RemoveContainer" containerID="5a471f74d21ea32dcabd7ee871d60d4c117a89430e258f313fc38f8b1e9d7beb"
Nov 03 22:15:02.517893 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:02.517727    5053 generic.go:356] "Generic (PLEG): container finished" podID="b5d77a67-7f9e-40ca-b67f-49b1be17239f" containerID="3fc5bd65e5c0df284cc5e1a5ef777adf36f4527d917e9098bb167d05ec8abdcc" exitCode=60
Nov 03 22:15:02.517893 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:02.517769    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms" event={"ID":"b5d77a67-7f9e-40ca-b67f-49b1be17239f","Type":"ContainerDied","Data":"3fc5bd65e5c0df284cc5e1a5ef777adf36f4527d917e9098bb167d05ec8abdcc"}
Nov 03 22:15:02.518166 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:02.518136    5053 scope.go:122] "RemoveContainer" containerID="3fc5bd65e5c0df284cc5e1a5ef777adf36f4527d917e9098bb167d05ec8abdcc"
Nov 03 22:15:02.521607 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:02.519851    5053 generic.go:356] "Generic (PLEG): container finished" podID="a83ccbc3-6590-43cd-bda1-1e8f478060c1" containerID="bd8cdc7b82a4f0a5f59f28a2931d1534cff74210c9cc2e92a7bc3fa32c510116" exitCode=60
Nov 03 22:15:02.521607 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:02.519884    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw" event={"ID":"a83ccbc3-6590-43cd-bda1-1e8f478060c1","Type":"ContainerDied","Data":"bd8cdc7b82a4f0a5f59f28a2931d1534cff74210c9cc2e92a7bc3fa32c510116"}
Nov 03 22:15:02.521607 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:02.520128    5053 scope.go:122] "RemoveContainer" containerID="bd8cdc7b82a4f0a5f59f28a2931d1534cff74210c9cc2e92a7bc3fa32c510116"
Nov 03 22:15:03.530526 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:03.530485    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms" event={"ID":"b5d77a67-7f9e-40ca-b67f-49b1be17239f","Type":"ContainerStarted","Data":"ae4ae41c9fddd662dbb824621c00896952385c9c0acd303fe299665e412d6f24"}
Nov 03 22:15:03.532678 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:03.532646    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw" event={"ID":"a83ccbc3-6590-43cd-bda1-1e8f478060c1","Type":"ContainerStarted","Data":"66be5619f4cfa28bdc48c31c876d3b8c9a5a2fd1cc1c711e9e5496b60b1a30c0"}
Nov 03 22:15:03.534928 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:03.534897    5053 generic.go:356] "Generic (PLEG): container finished" podID="f5db8aaf-b362-4a27-b828-54655dff7ad1" containerID="783df02839cc044a645c4dccb1b65b3530b9ff8728fdb684c97915f26c1576f8" exitCode=60
Nov 03 22:15:03.535070 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:03.534944    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt" event={"ID":"f5db8aaf-b362-4a27-b828-54655dff7ad1","Type":"ContainerDied","Data":"783df02839cc044a645c4dccb1b65b3530b9ff8728fdb684c97915f26c1576f8"}
Nov 03 22:15:03.535070 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:03.534998    5053 scope.go:122] "RemoveContainer" containerID="5ff7fabf7740c8b1b6d73108f0e913adbabb7bb33a4f4c091ae73f41b93e1156"
Nov 03 22:15:03.535325 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:03.535308    5053 scope.go:122] "RemoveContainer" containerID="783df02839cc044a645c4dccb1b65b3530b9ff8728fdb684c97915f26c1576f8"
Nov 03 22:15:03.535710 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:15:03.535686    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt_odh-ai-gateway-infra(f5db8aaf-b362-4a27-b828-54655dff7ad1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt" podUID="f5db8aaf-b362-4a27-b828-54655dff7ad1"
Nov 03 22:15:03.537053 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:03.537029    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj" event={"ID":"94f8d07a-8d67-4366-97b2-2773315a8ec0","Type":"ContainerStarted","Data":"8fc1d8394eeddae8c30edf882a88d57f3a3e9be46525b9a3fc8bedb0605aa2db"}
Nov 03 22:15:03.644590 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:03.644529    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms" podStartSLOduration=3.644509901 podStartE2EDuration="3.644509901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:03.64152741 +0000 UTC m=+3618.135449288" watchObservedRunningTime="2026-11-03 22:15:03.644509901 +0000 UTC m=+3618.138431777"
Nov 03 22:15:04.547744 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:04.547704    5053 generic.go:356] "Generic (PLEG): container finished" podID="a83ccbc3-6590-43cd-bda1-1e8f478060c1" containerID="66be5619f4cfa28bdc48c31c876d3b8c9a5a2fd1cc1c711e9e5496b60b1a30c0" exitCode=60
Nov 03 22:15:04.548220 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:04.547758    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw" event={"ID":"a83ccbc3-6590-43cd-bda1-1e8f478060c1","Type":"ContainerDied","Data":"66be5619f4cfa28bdc48c31c876d3b8c9a5a2fd1cc1c711e9e5496b60b1a30c0"}
Nov 03 22:15:04.548220 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:04.547818    5053 scope.go:122] "RemoveContainer" containerID="bd8cdc7b82a4f0a5f59f28a2931d1534cff74210c9cc2e92a7bc3fa32c510116"
Nov 03 22:15:04.548220 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:04.548131    5053 scope.go:122] "RemoveContainer" containerID="66be5619f4cfa28bdc48c31c876d3b8c9a5a2fd1cc1c711e9e5496b60b1a30c0"
Nov 03 22:15:04.548577 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:15:04.548527    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw_odh-ai-gateway-infra(a83ccbc3-6590-43cd-bda1-1e8f478060c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw" podUID="a83ccbc3-6590-43cd-bda1-1e8f478060c1"
Nov 03 22:15:04.550334 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:04.550312    5053 scope.go:122] "RemoveContainer" containerID="783df02839cc044a645c4dccb1b65b3530b9ff8728fdb684c97915f26c1576f8"
Nov 03 22:15:04.550673 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:15:04.550646    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt_odh-ai-gateway-infra(f5db8aaf-b362-4a27-b828-54655dff7ad1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt" podUID="f5db8aaf-b362-4a27-b828-54655dff7ad1"
Nov 03 22:15:04.552144 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:04.552122    5053 generic.go:356] "Generic (PLEG): container finished" podID="94f8d07a-8d67-4366-97b2-2773315a8ec0" containerID="8fc1d8394eeddae8c30edf882a88d57f3a3e9be46525b9a3fc8bedb0605aa2db" exitCode=60
Nov 03 22:15:04.552339 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:04.552149    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj" event={"ID":"94f8d07a-8d67-4366-97b2-2773315a8ec0","Type":"ContainerDied","Data":"8fc1d8394eeddae8c30edf882a88d57f3a3e9be46525b9a3fc8bedb0605aa2db"}
Nov 03 22:15:04.552459 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:04.552434    5053 scope.go:122] "RemoveContainer" containerID="8fc1d8394eeddae8c30edf882a88d57f3a3e9be46525b9a3fc8bedb0605aa2db"
Nov 03 22:15:04.552741 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:15:04.552716    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895735-jd7qj_odh-ai-gateway-infra(94f8d07a-8d67-4366-97b2-2773315a8ec0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj" podUID="94f8d07a-8d67-4366-97b2-2773315a8ec0"
Nov 03 22:15:04.554392 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:04.554366    5053 generic.go:356] "Generic (PLEG): container finished" podID="b5d77a67-7f9e-40ca-b67f-49b1be17239f" containerID="ae4ae41c9fddd662dbb824621c00896952385c9c0acd303fe299665e412d6f24" exitCode=60
Nov 03 22:15:04.554504 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:04.554405    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms" event={"ID":"b5d77a67-7f9e-40ca-b67f-49b1be17239f","Type":"ContainerDied","Data":"ae4ae41c9fddd662dbb824621c00896952385c9c0acd303fe299665e412d6f24"}
Nov 03 22:15:04.554683 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:04.554668    5053 scope.go:122] "RemoveContainer" containerID="ae4ae41c9fddd662dbb824621c00896952385c9c0acd303fe299665e412d6f24"
Nov 03 22:15:04.554914 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:15:04.554897    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms_odh-ai-gateway-infra(b5d77a67-7f9e-40ca-b67f-49b1be17239f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms" podUID="b5d77a67-7f9e-40ca-b67f-49b1be17239f"
Nov 03 22:15:04.561954 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:04.561934    5053 scope.go:122] "RemoveContainer" containerID="5a471f74d21ea32dcabd7ee871d60d4c117a89430e258f313fc38f8b1e9d7beb"
Nov 03 22:15:04.586960 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:04.586930    5053 scope.go:122] "RemoveContainer" containerID="3fc5bd65e5c0df284cc5e1a5ef777adf36f4527d917e9098bb167d05ec8abdcc"
Nov 03 22:15:05.564531 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:05.564500    5053 scope.go:122] "RemoveContainer" containerID="8fc1d8394eeddae8c30edf882a88d57f3a3e9be46525b9a3fc8bedb0605aa2db"
Nov 03 22:15:05.564961 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:15:05.564884    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895735-jd7qj_odh-ai-gateway-infra(94f8d07a-8d67-4366-97b2-2773315a8ec0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj" podUID="94f8d07a-8d67-4366-97b2-2773315a8ec0"
Nov 03 22:15:05.566126 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:05.566100    5053 scope.go:122] "RemoveContainer" containerID="ae4ae41c9fddd662dbb824621c00896952385c9c0acd303fe299665e412d6f24"
Nov 03 22:15:05.566459 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:15:05.566435    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms_odh-ai-gateway-infra(b5d77a67-7f9e-40ca-b67f-49b1be17239f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms" podUID="b5d77a67-7f9e-40ca-b67f-49b1be17239f"
Nov 03 22:15:05.567573 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:05.567559    5053 scope.go:122] "RemoveContainer" containerID="66be5619f4cfa28bdc48c31c876d3b8c9a5a2fd1cc1c711e9e5496b60b1a30c0"
Nov 03 22:15:05.567787 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:15:05.567770    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw_odh-ai-gateway-infra(a83ccbc3-6590-43cd-bda1-1e8f478060c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw" podUID="a83ccbc3-6590-43cd-bda1-1e8f478060c1"
Nov 03 22:15:10.929872 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:15:10.929815    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:15:13.142989 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.142951    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"]
Nov 03 22:15:13.149698 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.149672    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.191914 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.191861    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-workload-certs\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.192098 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.191998    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/886c2db8-998e-4425-a064-630afe2da8ca-istiod-ca-cert\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.192098 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.192045    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-istio-data\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.192098 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.192075    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m98fn\" (UniqueName: \"kubernetes.io/projected/886c2db8-998e-4425-a064-630afe2da8ca-kube-api-access-m98fn\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.192346 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.192216    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/886c2db8-998e-4425-a064-630afe2da8ca-istio-podinfo\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.192415 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.192348    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-workload-socket\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.192415 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.192380    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/886c2db8-998e-4425-a064-630afe2da8ca-istio-token\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.192531 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.192427    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-credential-socket\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.192531 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.192459    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-istio-envoy\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.222724 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.222688    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-534c77-openshift-default-dockercfg-c2gcp\""
Nov 03 22:15:13.279905 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.279861    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"]
Nov 03 22:15:13.296264 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.294057    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/886c2db8-998e-4425-a064-630afe2da8ca-istiod-ca-cert\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.296264 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.294130    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-istio-data\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.296264 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.294178    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m98fn\" (UniqueName: \"kubernetes.io/projected/886c2db8-998e-4425-a064-630afe2da8ca-kube-api-access-m98fn\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.296264 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.294260    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/886c2db8-998e-4425-a064-630afe2da8ca-istio-podinfo\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.296264 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.294360    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-workload-socket\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.296264 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.294397    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/886c2db8-998e-4425-a064-630afe2da8ca-istio-token\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.296264 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.294467    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-credential-socket\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.296264 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.294494    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-istio-envoy\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.296264 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.294579    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-workload-certs\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.296264 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.295024    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-workload-certs\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.296264 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.295796    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/886c2db8-998e-4425-a064-630afe2da8ca-istiod-ca-cert\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.296264 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.296038    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-istio-data\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.297075 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.296981    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-credential-socket\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.299661 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.297569    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-workload-socket\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.320411 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.314498    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-istio-envoy\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.337325 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.334989    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/886c2db8-998e-4425-a064-630afe2da8ca-istio-podinfo\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.358121 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.358012    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m98fn\" (UniqueName: \"kubernetes.io/projected/886c2db8-998e-4425-a064-630afe2da8ca-kube-api-access-m98fn\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.390294 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.390265    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/886c2db8-998e-4425-a064-630afe2da8ca-istio-token\") pod \"e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") " pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.463863 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.463830    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:13.903622 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.903589    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"]
Nov 03 22:15:13.903873 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:15:13.903850    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod886c2db8_998e_4425_a064_630afe2da8ca.slice/crio-d2a00bcdcf2c5c5d825edaf4ffe207d846525f3fd4369a5a6670c6785f4914b4 WatchSource:0}: Error finding container d2a00bcdcf2c5c5d825edaf4ffe207d846525f3fd4369a5a6670c6785f4914b4: Status 404 returned error can't find the container with id d2a00bcdcf2c5c5d825edaf4ffe207d846525f3fd4369a5a6670c6785f4914b4
Nov 03 22:15:13.906830 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.906765    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:15:13.906950 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.906845    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:15:13.906950 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:13.906887    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:15:14.638199 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:14.638147    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689" event={"ID":"886c2db8-998e-4425-a064-630afe2da8ca","Type":"ContainerStarted","Data":"81b9d9965618291f44b9f9b6b9b57a75adf2913ca0cfd05380f66caaed42325d"}
Nov 03 22:15:14.638670 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:14.638226    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689" event={"ID":"886c2db8-998e-4425-a064-630afe2da8ca","Type":"ContainerStarted","Data":"d2a00bcdcf2c5c5d825edaf4ffe207d846525f3fd4369a5a6670c6785f4914b4"}
Nov 03 22:15:14.660748 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:14.660718    5053 scope.go:122] "RemoveContainer" containerID="783df02839cc044a645c4dccb1b65b3530b9ff8728fdb684c97915f26c1576f8"
Nov 03 22:15:14.660981 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:14.660965    5053 scope.go:122] "RemoveContainer" containerID="ae4ae41c9fddd662dbb824621c00896952385c9c0acd303fe299665e412d6f24"
Nov 03 22:15:14.844733 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:14.844680    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689" podStartSLOduration=1.844665394 podStartE2EDuration="1.844665394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:14.843586864 +0000 UTC m=+3629.337508740" watchObservedRunningTime="2026-11-03 22:15:14.844665394 +0000 UTC m=+3629.338587269"
Nov 03 22:15:15.464944 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:15.464906    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:15.471118 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:15.471091    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:15.657275 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:15.650470    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt" event={"ID":"f5db8aaf-b362-4a27-b828-54655dff7ad1","Type":"ContainerStarted","Data":"0fb6d716c965c184bd1ecacffc77804d720e94970f0e9a816ed0d17808b62728"}
Nov 03 22:15:15.657275 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:15.652913    5053 generic.go:356] "Generic (PLEG): container finished" podID="b5d77a67-7f9e-40ca-b67f-49b1be17239f" containerID="cf1fe2416043c2eb345ccdfc6dbb5661c81c460cb55a16eb04e03bc579257197" exitCode=60
Nov 03 22:15:15.657275 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:15.652980    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms" event={"ID":"b5d77a67-7f9e-40ca-b67f-49b1be17239f","Type":"ContainerDied","Data":"cf1fe2416043c2eb345ccdfc6dbb5661c81c460cb55a16eb04e03bc579257197"}
Nov 03 22:15:15.657275 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:15.653048    5053 scope.go:122] "RemoveContainer" containerID="ae4ae41c9fddd662dbb824621c00896952385c9c0acd303fe299665e412d6f24"
Nov 03 22:15:15.657275 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:15.653366    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:15.657275 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:15.653533    5053 scope.go:122] "RemoveContainer" containerID="cf1fe2416043c2eb345ccdfc6dbb5661c81c460cb55a16eb04e03bc579257197"
Nov 03 22:15:15.657275 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:15:15.653790    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms_odh-ai-gateway-infra(b5d77a67-7f9e-40ca-b67f-49b1be17239f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms" podUID="b5d77a67-7f9e-40ca-b67f-49b1be17239f"
Nov 03 22:15:15.657275 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:15.654537    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:15:15.660044 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:15.660027    5053 scope.go:122] "RemoveContainer" containerID="8fc1d8394eeddae8c30edf882a88d57f3a3e9be46525b9a3fc8bedb0605aa2db"
Nov 03 22:15:15.764840 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:15.764785    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt" podStartSLOduration=15.764770917 podStartE2EDuration="15.764770917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:15.762411947 +0000 UTC m=+3630.256333820" watchObservedRunningTime="2026-11-03 22:15:15.764770917 +0000 UTC m=+3630.258692793"
Nov 03 22:15:16.663449 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:16.663416    5053 generic.go:356] "Generic (PLEG): container finished" podID="f5db8aaf-b362-4a27-b828-54655dff7ad1" containerID="0fb6d716c965c184bd1ecacffc77804d720e94970f0e9a816ed0d17808b62728" exitCode=60
Nov 03 22:15:16.663961 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:16.663486    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt" event={"ID":"f5db8aaf-b362-4a27-b828-54655dff7ad1","Type":"ContainerDied","Data":"0fb6d716c965c184bd1ecacffc77804d720e94970f0e9a816ed0d17808b62728"}
Nov 03 22:15:16.663961 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:16.663545    5053 scope.go:122] "RemoveContainer" containerID="783df02839cc044a645c4dccb1b65b3530b9ff8728fdb684c97915f26c1576f8"
Nov 03 22:15:16.664075 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:16.663962    5053 scope.go:122] "RemoveContainer" containerID="0fb6d716c965c184bd1ecacffc77804d720e94970f0e9a816ed0d17808b62728"
Nov 03 22:15:16.664397 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:15:16.664369    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt_odh-ai-gateway-infra(f5db8aaf-b362-4a27-b828-54655dff7ad1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt" podUID="f5db8aaf-b362-4a27-b828-54655dff7ad1"
Nov 03 22:15:16.665703 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:16.665673    5053 generic.go:356] "Generic (PLEG): container finished" podID="94f8d07a-8d67-4366-97b2-2773315a8ec0" containerID="be9d40ea6a92972f36d29bb71b5481fd715c07a8bc5de3413e437d94b04d1087" exitCode=60
Nov 03 22:15:16.665813 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:16.665751    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj" event={"ID":"94f8d07a-8d67-4366-97b2-2773315a8ec0","Type":"ContainerDied","Data":"be9d40ea6a92972f36d29bb71b5481fd715c07a8bc5de3413e437d94b04d1087"}
Nov 03 22:15:16.666167 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:16.666151    5053 scope.go:122] "RemoveContainer" containerID="be9d40ea6a92972f36d29bb71b5481fd715c07a8bc5de3413e437d94b04d1087"
Nov 03 22:15:16.666540 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:15:16.666520    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895735-jd7qj_odh-ai-gateway-infra(94f8d07a-8d67-4366-97b2-2773315a8ec0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj" podUID="94f8d07a-8d67-4366-97b2-2773315a8ec0"
Nov 03 22:15:16.699170 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:16.699144    5053 scope.go:122] "RemoveContainer" containerID="8fc1d8394eeddae8c30edf882a88d57f3a3e9be46525b9a3fc8bedb0605aa2db"
Nov 03 22:15:16.773583 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:16.773549    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt"]
Nov 03 22:15:17.001594 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:17.001558    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms"]
Nov 03 22:15:17.186464 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:17.186432    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms"
Nov 03 22:15:17.258721 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:17.258640    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d77a67-7f9e-40ca-b67f-49b1be17239f-kube-api-access-l6wqp\" (UniqueName: \"kubernetes.io/projected/b5d77a67-7f9e-40ca-b67f-49b1be17239f-kube-api-access-l6wqp\") pod \"b5d77a67-7f9e-40ca-b67f-49b1be17239f\" (UID: \"b5d77a67-7f9e-40ca-b67f-49b1be17239f\") "
Nov 03 22:15:17.258917 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:17.258903    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5d77a67-7f9e-40ca-b67f-49b1be17239f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5d77a67-7f9e-40ca-b67f-49b1be17239f-maas-api-service-ca\") pod \"b5d77a67-7f9e-40ca-b67f-49b1be17239f\" (UID: \"b5d77a67-7f9e-40ca-b67f-49b1be17239f\") "
Nov 03 22:15:17.259291 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:17.259228    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5d77a67-7f9e-40ca-b67f-49b1be17239f-maas-api-service-ca" pod "b5d77a67-7f9e-40ca-b67f-49b1be17239f" (UID: "b5d77a67-7f9e-40ca-b67f-49b1be17239f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:15:17.260871 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:17.260847    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d77a67-7f9e-40ca-b67f-49b1be17239f-kube-api-access-l6wqp" pod "b5d77a67-7f9e-40ca-b67f-49b1be17239f" (UID: "b5d77a67-7f9e-40ca-b67f-49b1be17239f"). InnerVolumeSpecName "kube-api-access-l6wqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:15:17.360198 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:17.360167    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5d77a67-7f9e-40ca-b67f-49b1be17239f-maas-api-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:17.360198 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:17.360196    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6wqp\" (UniqueName: \"kubernetes.io/projected/b5d77a67-7f9e-40ca-b67f-49b1be17239f-kube-api-access-l6wqp\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:17.659916 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:17.659824    5053 scope.go:122] "RemoveContainer" containerID="66be5619f4cfa28bdc48c31c876d3b8c9a5a2fd1cc1c711e9e5496b60b1a30c0"
Nov 03 22:15:17.684290 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:17.684262    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms" event={"ID":"b5d77a67-7f9e-40ca-b67f-49b1be17239f","Type":"ContainerDied","Data":"bf4f421d46d581907763d1940283ba465b4381a5904554e785f4a11947e440f1"}
Nov 03 22:15:17.684679 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:17.684333    5053 scope.go:122] "RemoveContainer" containerID="cf1fe2416043c2eb345ccdfc6dbb5661c81c460cb55a16eb04e03bc579257197"
Nov 03 22:15:17.684679 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:17.684272    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms"
Nov 03 22:15:17.859065 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:17.859039    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt"
Nov 03 22:15:17.921335 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:17.921158    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms"]
Nov 03 22:15:17.925055 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:17.925025    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895735-clpms"]
Nov 03 22:15:17.954843 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:17.954811    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj"]
Nov 03 22:15:17.968547 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:17.968510    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5db8aaf-b362-4a27-b828-54655dff7ad1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5db8aaf-b362-4a27-b828-54655dff7ad1-maas-api-service-ca\") pod \"f5db8aaf-b362-4a27-b828-54655dff7ad1\" (UID: \"f5db8aaf-b362-4a27-b828-54655dff7ad1\") "
Nov 03 22:15:17.968684 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:17.968579    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5db8aaf-b362-4a27-b828-54655dff7ad1-kube-api-access-nvrw6\" (UniqueName: \"kubernetes.io/projected/f5db8aaf-b362-4a27-b828-54655dff7ad1-kube-api-access-nvrw6\") pod \"f5db8aaf-b362-4a27-b828-54655dff7ad1\" (UID: \"f5db8aaf-b362-4a27-b828-54655dff7ad1\") "
Nov 03 22:15:17.969292 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:17.969262    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5db8aaf-b362-4a27-b828-54655dff7ad1-maas-api-service-ca" pod "f5db8aaf-b362-4a27-b828-54655dff7ad1" (UID: "f5db8aaf-b362-4a27-b828-54655dff7ad1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:15:17.971185 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:17.971154    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5db8aaf-b362-4a27-b828-54655dff7ad1-kube-api-access-nvrw6" pod "f5db8aaf-b362-4a27-b828-54655dff7ad1" (UID: "f5db8aaf-b362-4a27-b828-54655dff7ad1"). InnerVolumeSpecName "kube-api-access-nvrw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:15:18.070277 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.070223    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5db8aaf-b362-4a27-b828-54655dff7ad1-maas-api-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:18.070277 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.070279    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvrw6\" (UniqueName: \"kubernetes.io/projected/f5db8aaf-b362-4a27-b828-54655dff7ad1-kube-api-access-nvrw6\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:18.116704 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.116670    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj"
Nov 03 22:15:18.170890 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.170858    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94f8d07a-8d67-4366-97b2-2773315a8ec0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f8d07a-8d67-4366-97b2-2773315a8ec0-maas-api-service-ca\") pod \"94f8d07a-8d67-4366-97b2-2773315a8ec0\" (UID: \"94f8d07a-8d67-4366-97b2-2773315a8ec0\") "
Nov 03 22:15:18.171139 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.171106    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f8d07a-8d67-4366-97b2-2773315a8ec0-kube-api-access-8zsqd\" (UniqueName: \"kubernetes.io/projected/94f8d07a-8d67-4366-97b2-2773315a8ec0-kube-api-access-8zsqd\") pod \"94f8d07a-8d67-4366-97b2-2773315a8ec0\" (UID: \"94f8d07a-8d67-4366-97b2-2773315a8ec0\") "
Nov 03 22:15:18.171264 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.171196    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f8d07a-8d67-4366-97b2-2773315a8ec0-maas-api-service-ca" pod "94f8d07a-8d67-4366-97b2-2773315a8ec0" (UID: "94f8d07a-8d67-4366-97b2-2773315a8ec0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:15:18.171756 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.171657    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f8d07a-8d67-4366-97b2-2773315a8ec0-maas-api-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:18.173616 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.173593    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f8d07a-8d67-4366-97b2-2773315a8ec0-kube-api-access-8zsqd" pod "94f8d07a-8d67-4366-97b2-2773315a8ec0" (UID: "94f8d07a-8d67-4366-97b2-2773315a8ec0"). InnerVolumeSpecName "kube-api-access-8zsqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:15:18.215447 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.215405    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-12dee4-qqjjp"]
Nov 03 22:15:18.216533 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.216391    5053 cpu_manager.go:414] "containerMap: removing container" podUID="b5d77a67-7f9e-40ca-b67f-49b1be17239f" containerName="cleanup"
Nov 03 22:15:18.216533 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.216410    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d77a67-7f9e-40ca-b67f-49b1be17239f" containerName="cleanup"
Nov 03 22:15:18.216533 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.216422    5053 cpu_manager.go:414] "containerMap: removing container" podUID="f5db8aaf-b362-4a27-b828-54655dff7ad1" containerName="cleanup"
Nov 03 22:15:18.216533 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.216431    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5db8aaf-b362-4a27-b828-54655dff7ad1" containerName="cleanup"
Nov 03 22:15:18.216533 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.216445    5053 cpu_manager.go:414] "containerMap: removing container" podUID="f5db8aaf-b362-4a27-b828-54655dff7ad1" containerName="cleanup"
Nov 03 22:15:18.216533 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.216453    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5db8aaf-b362-4a27-b828-54655dff7ad1" containerName="cleanup"
Nov 03 22:15:18.216533 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.216475    5053 cpu_manager.go:414] "containerMap: removing container" podUID="94f8d07a-8d67-4366-97b2-2773315a8ec0" containerName="cleanup"
Nov 03 22:15:18.216533 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.216482    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f8d07a-8d67-4366-97b2-2773315a8ec0" containerName="cleanup"
Nov 03 22:15:18.216533 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.216510    5053 cpu_manager.go:414] "containerMap: removing container" podUID="b5d77a67-7f9e-40ca-b67f-49b1be17239f" containerName="cleanup"
Nov 03 22:15:18.216533 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.216517    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d77a67-7f9e-40ca-b67f-49b1be17239f" containerName="cleanup"
Nov 03 22:15:18.216533 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.216537    5053 cpu_manager.go:414] "containerMap: removing container" podUID="94f8d07a-8d67-4366-97b2-2773315a8ec0" containerName="cleanup"
Nov 03 22:15:18.216965 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.216544    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f8d07a-8d67-4366-97b2-2773315a8ec0" containerName="cleanup"
Nov 03 22:15:18.216965 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.216568    5053 cpu_manager.go:414] "containerMap: removing container" podUID="b5d77a67-7f9e-40ca-b67f-49b1be17239f" containerName="cleanup"
Nov 03 22:15:18.216965 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.216574    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d77a67-7f9e-40ca-b67f-49b1be17239f" containerName="cleanup"
Nov 03 22:15:18.216965 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.216743    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5db8aaf-b362-4a27-b828-54655dff7ad1" containerName="cleanup"
Nov 03 22:15:18.216965 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.216756    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f8d07a-8d67-4366-97b2-2773315a8ec0" containerName="cleanup"
Nov 03 22:15:18.216965 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.216767    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d77a67-7f9e-40ca-b67f-49b1be17239f" containerName="cleanup"
Nov 03 22:15:18.216965 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.216776    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f8d07a-8d67-4366-97b2-2773315a8ec0" containerName="cleanup"
Nov 03 22:15:18.216965 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.216785    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5db8aaf-b362-4a27-b828-54655dff7ad1" containerName="cleanup"
Nov 03 22:15:18.216965 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.216797    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d77a67-7f9e-40ca-b67f-49b1be17239f" containerName="cleanup"
Nov 03 22:15:18.216965 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.216807    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f8d07a-8d67-4366-97b2-2773315a8ec0" containerName="cleanup"
Nov 03 22:15:18.216965 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.216816    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d77a67-7f9e-40ca-b67f-49b1be17239f" containerName="cleanup"
Nov 03 22:15:18.221408 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.221363    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-12dee4-qqjjp"
Nov 03 22:15:18.273746 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.273688    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762459e3-f8a0-40f3-bd1c-24eaa187d79a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-12dee4-qqjjp\" (UID: \"762459e3-f8a0-40f3-bd1c-24eaa187d79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-12dee4-qqjjp"
Nov 03 22:15:18.273944 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.273753    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762459e3-f8a0-40f3-bd1c-24eaa187d79a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-12dee4-qqjjp\" (UID: \"762459e3-f8a0-40f3-bd1c-24eaa187d79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-12dee4-qqjjp"
Nov 03 22:15:18.273944 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.273906    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbcbq\" (UniqueName: \"kubernetes.io/projected/762459e3-f8a0-40f3-bd1c-24eaa187d79a-kube-api-access-zbcbq\") pod \"maas-api-revoke-keys-e2e-autores-12dee4-qqjjp\" (UID: \"762459e3-f8a0-40f3-bd1c-24eaa187d79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-12dee4-qqjjp"
Nov 03 22:15:18.274216 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.274193    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zsqd\" (UniqueName: \"kubernetes.io/projected/94f8d07a-8d67-4366-97b2-2773315a8ec0-kube-api-access-8zsqd\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:18.277592 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.277566    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-12dee4-qqjjp"]
Nov 03 22:15:18.361553 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.361523    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:15:18.375344 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.375304    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762459e3-f8a0-40f3-bd1c-24eaa187d79a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-12dee4-qqjjp\" (UID: \"762459e3-f8a0-40f3-bd1c-24eaa187d79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-12dee4-qqjjp"
Nov 03 22:15:18.375546 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.375406    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762459e3-f8a0-40f3-bd1c-24eaa187d79a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-12dee4-qqjjp\" (UID: \"762459e3-f8a0-40f3-bd1c-24eaa187d79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-12dee4-qqjjp"
Nov 03 22:15:18.375619 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.375562    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbcbq\" (UniqueName: \"kubernetes.io/projected/762459e3-f8a0-40f3-bd1c-24eaa187d79a-kube-api-access-zbcbq\") pod \"maas-api-revoke-keys-e2e-autores-12dee4-qqjjp\" (UID: \"762459e3-f8a0-40f3-bd1c-24eaa187d79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-12dee4-qqjjp"
Nov 03 22:15:18.376187 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.376169    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762459e3-f8a0-40f3-bd1c-24eaa187d79a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-12dee4-qqjjp\" (UID: \"762459e3-f8a0-40f3-bd1c-24eaa187d79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-12dee4-qqjjp"
Nov 03 22:15:18.378036 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.377995    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762459e3-f8a0-40f3-bd1c-24eaa187d79a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-12dee4-qqjjp\" (UID: \"762459e3-f8a0-40f3-bd1c-24eaa187d79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-12dee4-qqjjp"
Nov 03 22:15:18.433138 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.433058    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbcbq\" (UniqueName: \"kubernetes.io/projected/762459e3-f8a0-40f3-bd1c-24eaa187d79a-kube-api-access-zbcbq\") pod \"maas-api-revoke-keys-e2e-autores-12dee4-qqjjp\" (UID: \"762459e3-f8a0-40f3-bd1c-24eaa187d79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-12dee4-qqjjp"
Nov 03 22:15:18.539216 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.539176    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-12dee4-qqjjp"
Nov 03 22:15:18.699761 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.699719    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt" event={"ID":"f5db8aaf-b362-4a27-b828-54655dff7ad1","Type":"ContainerDied","Data":"fd8f35c6c6347a280283193cfa956e07b0b712f81836f12e3f778181828934e9"}
Nov 03 22:15:18.700330 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.699797    5053 scope.go:122] "RemoveContainer" containerID="0fb6d716c965c184bd1ecacffc77804d720e94970f0e9a816ed0d17808b62728"
Nov 03 22:15:18.700330 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.699940    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt"
Nov 03 22:15:18.710291 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.710255    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj" event={"ID":"94f8d07a-8d67-4366-97b2-2773315a8ec0","Type":"ContainerDied","Data":"b9152a4f4d020d59b49c1f4e946448eabfac0ed7cbaf989bf1d90c245005f8b9"}
Nov 03 22:15:18.710590 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.710366    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj"
Nov 03 22:15:18.715471 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.715395    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw" event={"ID":"a83ccbc3-6590-43cd-bda1-1e8f478060c1","Type":"ContainerStarted","Data":"5f47d8cffe0df1beffdc5354475c6aa3089cfd57cd2a1ae4ee64afd239b2da44"}
Nov 03 22:15:18.726108 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.726085    5053 scope.go:122] "RemoveContainer" containerID="be9d40ea6a92972f36d29bb71b5481fd715c07a8bc5de3413e437d94b04d1087"
Nov 03 22:15:18.948182 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:15:18.946958    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762459e3_f8a0_40f3_bd1c_24eaa187d79a.slice/crio-8fce86fab6c3a36ee228ae37f94b1757a52e77e3e0bf382603b736722e3027af WatchSource:0}: Error finding container 8fce86fab6c3a36ee228ae37f94b1757a52e77e3e0bf382603b736722e3027af: Status 404 returned error can't find the container with id 8fce86fab6c3a36ee228ae37f94b1757a52e77e3e0bf382603b736722e3027af
Nov 03 22:15:18.948182 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:18.947024    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-12dee4-qqjjp"]
Nov 03 22:15:19.003509 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:19.003478    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt"]
Nov 03 22:15:19.014144 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:19.014115    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-12dee4-29895735-bnpkt"]
Nov 03 22:15:19.121434 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:19.121390    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj"]
Nov 03 22:15:19.127489 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:19.127458    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895735-jd7qj"]
Nov 03 22:15:19.668660 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:19.668576    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94f8d07a-8d67-4366-97b2-2773315a8ec0" path="/var/lib/kubelet/pods/94f8d07a-8d67-4366-97b2-2773315a8ec0/volumes"
Nov 03 22:15:19.669162 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:19.669139    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5d77a67-7f9e-40ca-b67f-49b1be17239f" path="/var/lib/kubelet/pods/b5d77a67-7f9e-40ca-b67f-49b1be17239f/volumes"
Nov 03 22:15:19.669916 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:19.669895    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5db8aaf-b362-4a27-b828-54655dff7ad1" path="/var/lib/kubelet/pods/f5db8aaf-b362-4a27-b828-54655dff7ad1/volumes"
Nov 03 22:15:19.725364 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:19.725316    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-12dee4-qqjjp" event={"ID":"762459e3-f8a0-40f3-bd1c-24eaa187d79a","Type":"ContainerStarted","Data":"28a0f94548113edfe9d8c36efebbaf36a386ccb97ee9b80b6640090ab3fda6cb"}
Nov 03 22:15:19.725858 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:19.725396    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-12dee4-qqjjp" event={"ID":"762459e3-f8a0-40f3-bd1c-24eaa187d79a","Type":"ContainerStarted","Data":"8fce86fab6c3a36ee228ae37f94b1757a52e77e3e0bf382603b736722e3027af"}
Nov 03 22:15:19.729828 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:19.729805    5053 generic.go:356] "Generic (PLEG): container finished" podID="a83ccbc3-6590-43cd-bda1-1e8f478060c1" containerID="5f47d8cffe0df1beffdc5354475c6aa3089cfd57cd2a1ae4ee64afd239b2da44" exitCode=60
Nov 03 22:15:19.729958 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:19.729889    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw" event={"ID":"a83ccbc3-6590-43cd-bda1-1e8f478060c1","Type":"ContainerDied","Data":"5f47d8cffe0df1beffdc5354475c6aa3089cfd57cd2a1ae4ee64afd239b2da44"}
Nov 03 22:15:19.729958 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:19.729952    5053 scope.go:122] "RemoveContainer" containerID="66be5619f4cfa28bdc48c31c876d3b8c9a5a2fd1cc1c711e9e5496b60b1a30c0"
Nov 03 22:15:19.730433 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:19.730418    5053 scope.go:122] "RemoveContainer" containerID="5f47d8cffe0df1beffdc5354475c6aa3089cfd57cd2a1ae4ee64afd239b2da44"
Nov 03 22:15:19.730789 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:15:19.730762    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw_odh-ai-gateway-infra(a83ccbc3-6590-43cd-bda1-1e8f478060c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw" podUID="a83ccbc3-6590-43cd-bda1-1e8f478060c1"
Nov 03 22:15:19.854790 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:19.854725    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-12dee4-qqjjp" podStartSLOduration=1.854706902 podStartE2EDuration="1.854706902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:19.840278995 +0000 UTC m=+3634.334200869" watchObservedRunningTime="2026-11-03 22:15:19.854706902 +0000 UTC m=+3634.348628780"
Nov 03 22:15:19.897503 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:19.897474    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw"]
Nov 03 22:15:20.739597 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:20.739565    5053 generic.go:356] "Generic (PLEG): container finished" podID="762459e3-f8a0-40f3-bd1c-24eaa187d79a" containerID="28a0f94548113edfe9d8c36efebbaf36a386ccb97ee9b80b6640090ab3fda6cb" exitCode=0
Nov 03 22:15:20.740126 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:20.739703    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-12dee4-qqjjp" event={"ID":"762459e3-f8a0-40f3-bd1c-24eaa187d79a","Type":"ContainerDied","Data":"28a0f94548113edfe9d8c36efebbaf36a386ccb97ee9b80b6640090ab3fda6cb"}
Nov 03 22:15:20.936488 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:20.936452    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw"
Nov 03 22:15:21.029318 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:21.028832    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83ccbc3-6590-43cd-bda1-1e8f478060c1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83ccbc3-6590-43cd-bda1-1e8f478060c1-maas-api-service-ca\") pod \"a83ccbc3-6590-43cd-bda1-1e8f478060c1\" (UID: \"a83ccbc3-6590-43cd-bda1-1e8f478060c1\") "
Nov 03 22:15:21.029318 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:21.029002    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83ccbc3-6590-43cd-bda1-1e8f478060c1-kube-api-access-7r9xx\" (UniqueName: \"kubernetes.io/projected/a83ccbc3-6590-43cd-bda1-1e8f478060c1-kube-api-access-7r9xx\") pod \"a83ccbc3-6590-43cd-bda1-1e8f478060c1\" (UID: \"a83ccbc3-6590-43cd-bda1-1e8f478060c1\") "
Nov 03 22:15:21.055637 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:21.055593    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83ccbc3-6590-43cd-bda1-1e8f478060c1-maas-api-service-ca" pod "a83ccbc3-6590-43cd-bda1-1e8f478060c1" (UID: "a83ccbc3-6590-43cd-bda1-1e8f478060c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:15:21.075419 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:21.075378    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83ccbc3-6590-43cd-bda1-1e8f478060c1-kube-api-access-7r9xx" pod "a83ccbc3-6590-43cd-bda1-1e8f478060c1" (UID: "a83ccbc3-6590-43cd-bda1-1e8f478060c1"). InnerVolumeSpecName "kube-api-access-7r9xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:15:21.130780 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:21.130737    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83ccbc3-6590-43cd-bda1-1e8f478060c1-maas-api-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:21.130780 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:21.130781    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r9xx\" (UniqueName: \"kubernetes.io/projected/a83ccbc3-6590-43cd-bda1-1e8f478060c1-kube-api-access-7r9xx\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:21.750058 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:21.750020    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw"
Nov 03 22:15:21.750555 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:21.750056    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw" event={"ID":"a83ccbc3-6590-43cd-bda1-1e8f478060c1","Type":"ContainerDied","Data":"728b1ce7e5b4146888f97be26db76f2ef83bf14644e50190d2af425c43025625"}
Nov 03 22:15:21.750555 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:21.750115    5053 scope.go:122] "RemoveContainer" containerID="5f47d8cffe0df1beffdc5354475c6aa3089cfd57cd2a1ae4ee64afd239b2da44"
Nov 03 22:15:21.915039 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:21.915008    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw"]
Nov 03 22:15:21.918768 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:21.918741    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fb477-29895735-bjwvw"]
Nov 03 22:15:21.946608 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:21.946584    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-12dee4-qqjjp"
Nov 03 22:15:22.042948 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:22.042907    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/762459e3-f8a0-40f3-bd1c-24eaa187d79a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762459e3-f8a0-40f3-bd1c-24eaa187d79a-maas-api-xks-ca\") pod \"762459e3-f8a0-40f3-bd1c-24eaa187d79a\" (UID: \"762459e3-f8a0-40f3-bd1c-24eaa187d79a\") "
Nov 03 22:15:22.043135 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:22.043089    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/762459e3-f8a0-40f3-bd1c-24eaa187d79a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762459e3-f8a0-40f3-bd1c-24eaa187d79a-maas-api-openshift-ca\") pod \"762459e3-f8a0-40f3-bd1c-24eaa187d79a\" (UID: \"762459e3-f8a0-40f3-bd1c-24eaa187d79a\") "
Nov 03 22:15:22.043219 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:22.043178    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762459e3-f8a0-40f3-bd1c-24eaa187d79a-kube-api-access-zbcbq\" (UniqueName: \"kubernetes.io/projected/762459e3-f8a0-40f3-bd1c-24eaa187d79a-kube-api-access-zbcbq\") pod \"762459e3-f8a0-40f3-bd1c-24eaa187d79a\" (UID: \"762459e3-f8a0-40f3-bd1c-24eaa187d79a\") "
Nov 03 22:15:22.043709 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:22.043681    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/762459e3-f8a0-40f3-bd1c-24eaa187d79a-maas-api-openshift-ca" pod "762459e3-f8a0-40f3-bd1c-24eaa187d79a" (UID: "762459e3-f8a0-40f3-bd1c-24eaa187d79a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:15:22.045898 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:22.045870    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762459e3-f8a0-40f3-bd1c-24eaa187d79a-kube-api-access-zbcbq" pod "762459e3-f8a0-40f3-bd1c-24eaa187d79a" (UID: "762459e3-f8a0-40f3-bd1c-24eaa187d79a"). InnerVolumeSpecName "kube-api-access-zbcbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:15:22.046077 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:22.046053    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/762459e3-f8a0-40f3-bd1c-24eaa187d79a-maas-api-xks-ca" pod "762459e3-f8a0-40f3-bd1c-24eaa187d79a" (UID: "762459e3-f8a0-40f3-bd1c-24eaa187d79a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:15:22.145052 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:22.145017    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762459e3-f8a0-40f3-bd1c-24eaa187d79a-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:22.145052 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:22.145049    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbcbq\" (UniqueName: \"kubernetes.io/projected/762459e3-f8a0-40f3-bd1c-24eaa187d79a-kube-api-access-zbcbq\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:22.145052 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:22.145060    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762459e3-f8a0-40f3-bd1c-24eaa187d79a-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:22.759620 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:22.759588    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-12dee4-qqjjp" event={"ID":"762459e3-f8a0-40f3-bd1c-24eaa187d79a","Type":"ContainerDied","Data":"8fce86fab6c3a36ee228ae37f94b1757a52e77e3e0bf382603b736722e3027af"}
Nov 03 22:15:22.759620 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:22.759612    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-12dee4-qqjjp"
Nov 03 22:15:22.760078 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:22.759634    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fce86fab6c3a36ee228ae37f94b1757a52e77e3e0bf382603b736722e3027af"
Nov 03 22:15:23.668563 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:23.668534    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a83ccbc3-6590-43cd-bda1-1e8f478060c1" path="/var/lib/kubelet/pods/a83ccbc3-6590-43cd-bda1-1e8f478060c1/volumes"
Nov 03 22:15:40.930533 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:15:40.930493    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:15:41.736554 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:41.736517    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"]
Nov 03 22:15:46.151707 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:46.151679    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"]
Nov 03 22:15:46.152126 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:46.151892    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm" podUID="79ff394d-d937-46b1-b994-e180a924a3e0" containerName="maas-api" containerID="cri-o://81b7126a261fb1605a2c7cd2b5de5cb407bdfd34dfa06f0e90ce1f51298ee76e" gracePeriod=30
Nov 03 22:15:46.461444 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:46.461418    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"
Nov 03 22:15:46.584593 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:46.584551    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79ff394d-d937-46b1-b994-e180a924a3e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ff394d-d937-46b1-b994-e180a924a3e0-openshift-service-ca\") pod \"79ff394d-d937-46b1-b994-e180a924a3e0\" (UID: \"79ff394d-d937-46b1-b994-e180a924a3e0\") "
Nov 03 22:15:46.584777 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:46.584694    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79ff394d-d937-46b1-b994-e180a924a3e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ff394d-d937-46b1-b994-e180a924a3e0-maas-api-tls\") pod \"79ff394d-d937-46b1-b994-e180a924a3e0\" (UID: \"79ff394d-d937-46b1-b994-e180a924a3e0\") "
Nov 03 22:15:46.584777 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:46.584751    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79ff394d-d937-46b1-b994-e180a924a3e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ff394d-d937-46b1-b994-e180a924a3e0-metrics-certs\") pod \"79ff394d-d937-46b1-b994-e180a924a3e0\" (UID: \"79ff394d-d937-46b1-b994-e180a924a3e0\") "
Nov 03 22:15:46.584898 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:46.584886    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ff394d-d937-46b1-b994-e180a924a3e0-kube-api-access-h4vs9\" (UniqueName: \"kubernetes.io/projected/79ff394d-d937-46b1-b994-e180a924a3e0-kube-api-access-h4vs9\") pod \"79ff394d-d937-46b1-b994-e180a924a3e0\" (UID: \"79ff394d-d937-46b1-b994-e180a924a3e0\") "
Nov 03 22:15:46.584975 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:46.584949    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ff394d-d937-46b1-b994-e180a924a3e0-openshift-service-ca" pod "79ff394d-d937-46b1-b994-e180a924a3e0" (UID: "79ff394d-d937-46b1-b994-e180a924a3e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:15:46.585464 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:46.585427    5053 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ff394d-d937-46b1-b994-e180a924a3e0-openshift-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:46.590322 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:46.590288    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79ff394d-d937-46b1-b994-e180a924a3e0-maas-api-tls" pod "79ff394d-d937-46b1-b994-e180a924a3e0" (UID: "79ff394d-d937-46b1-b994-e180a924a3e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:15:46.591438 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:46.591409    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ff394d-d937-46b1-b994-e180a924a3e0-kube-api-access-h4vs9" pod "79ff394d-d937-46b1-b994-e180a924a3e0" (UID: "79ff394d-d937-46b1-b994-e180a924a3e0"). InnerVolumeSpecName "kube-api-access-h4vs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:15:46.591796 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:46.591768    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79ff394d-d937-46b1-b994-e180a924a3e0-metrics-certs" pod "79ff394d-d937-46b1-b994-e180a924a3e0" (UID: "79ff394d-d937-46b1-b994-e180a924a3e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:15:46.691607 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:46.689301    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ff394d-d937-46b1-b994-e180a924a3e0-maas-api-tls\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:46.691607 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:46.689346    5053 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ff394d-d937-46b1-b994-e180a924a3e0-metrics-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:46.691607 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:46.689360    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4vs9\" (UniqueName: \"kubernetes.io/projected/79ff394d-d937-46b1-b994-e180a924a3e0-kube-api-access-h4vs9\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:46.934189 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:46.934154    5053 generic.go:356] "Generic (PLEG): container finished" podID="79ff394d-d937-46b1-b994-e180a924a3e0" containerID="81b7126a261fb1605a2c7cd2b5de5cb407bdfd34dfa06f0e90ce1f51298ee76e" exitCode=0
Nov 03 22:15:46.934439 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:46.934203    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"
Nov 03 22:15:46.934439 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:46.934257    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm" event={"ID":"79ff394d-d937-46b1-b994-e180a924a3e0","Type":"ContainerDied","Data":"81b7126a261fb1605a2c7cd2b5de5cb407bdfd34dfa06f0e90ce1f51298ee76e"}
Nov 03 22:15:46.934439 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:46.934319    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm" event={"ID":"79ff394d-d937-46b1-b994-e180a924a3e0","Type":"ContainerDied","Data":"443ff47084f2cdf1c4352d984bfc0ee39825c0d2b9faf9231847b07cb9a898c0"}
Nov 03 22:15:46.934439 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:46.934347    5053 scope.go:122] "RemoveContainer" containerID="81b7126a261fb1605a2c7cd2b5de5cb407bdfd34dfa06f0e90ce1f51298ee76e"
Nov 03 22:15:46.946196 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:46.946179    5053 scope.go:122] "RemoveContainer" containerID="81b7126a261fb1605a2c7cd2b5de5cb407bdfd34dfa06f0e90ce1f51298ee76e"
Nov 03 22:15:46.946517 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:15:46.946497    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b7126a261fb1605a2c7cd2b5de5cb407bdfd34dfa06f0e90ce1f51298ee76e\": container with ID starting with 81b7126a261fb1605a2c7cd2b5de5cb407bdfd34dfa06f0e90ce1f51298ee76e not found: ID does not exist" containerID="81b7126a261fb1605a2c7cd2b5de5cb407bdfd34dfa06f0e90ce1f51298ee76e"
Nov 03 22:15:46.946593 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:46.946525    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b7126a261fb1605a2c7cd2b5de5cb407bdfd34dfa06f0e90ce1f51298ee76e"} err="failed to get container status \"81b7126a261fb1605a2c7cd2b5de5cb407bdfd34dfa06f0e90ce1f51298ee76e\": rpc error: code = NotFound desc = could not find container \"81b7126a261fb1605a2c7cd2b5de5cb407bdfd34dfa06f0e90ce1f51298ee76e\": container with ID starting with 81b7126a261fb1605a2c7cd2b5de5cb407bdfd34dfa06f0e90ce1f51298ee76e not found: ID does not exist"
Nov 03 22:15:47.119262 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:47.105323    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"]
Nov 03 22:15:47.119262 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:47.112922    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-12dee4-bc677c5f4-mqtqm"]
Nov 03 22:15:47.673458 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:47.673417    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79ff394d-d937-46b1-b994-e180a924a3e0" path="/var/lib/kubelet/pods/79ff394d-d937-46b1-b994-e180a924a3e0/volumes"
Nov 03 22:15:48.335856 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:48.335820    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-12dee4-qqjjp"]
Nov 03 22:15:48.384718 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:48.384686    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-12dee4-qqjjp"]
Nov 03 22:15:49.667975 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:49.667942    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="762459e3-f8a0-40f3-bd1c-24eaa187d79a" path="/var/lib/kubelet/pods/762459e3-f8a0-40f3-bd1c-24eaa187d79a/volumes"
Nov 03 22:15:56.757977 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:56.757923    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr" podUID="9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9" containerName="main" containerID="cri-o://239708540b812c4ef9cc3a211b873d04327cdc43a273a344002f92710003f619" gracePeriod=45
Nov 03 22:15:57.008791 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:57.008705    5053 generic.go:356] "Generic (PLEG): container finished" podID="9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9" containerID="239708540b812c4ef9cc3a211b873d04327cdc43a273a344002f92710003f619" exitCode=0
Nov 03 22:15:57.008945 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:57.008781    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr" event={"ID":"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9","Type":"ContainerDied","Data":"239708540b812c4ef9cc3a211b873d04327cdc43a273a344002f92710003f619"}
Nov 03 22:15:57.049687 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:57.049658    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:15:57.136099 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:57.136067    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-dshm\") pod \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\" (UID: \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\") "
Nov 03 22:15:57.136309 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:57.136111    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-home\" (UniqueName: \"kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-home\") pod \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\" (UID: \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\") "
Nov 03 22:15:57.136309 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:57.136149    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-tls-certs\") pod \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\" (UID: \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\") "
Nov 03 22:15:57.136309 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:57.136292    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-model-cache\") pod \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\" (UID: \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\") "
Nov 03 22:15:57.136497 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:57.136407    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-tmp-dir\") pod \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\" (UID: \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\") "
Nov 03 22:15:57.136497 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:57.136471    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-home" pod "9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9" (UID: "9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:15:57.136613 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:57.136478    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-kube-api-access-9lxxq\" (UniqueName: \"kubernetes.io/projected/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-kube-api-access-9lxxq\") pod \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\" (UID: \"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9\") "
Nov 03 22:15:57.136762 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:57.136707    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-tmp-dir" pod "9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9" (UID: "9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:15:57.136762 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:57.136739    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-model-cache" pod "9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9" (UID: "9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:15:57.137487 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:57.137458    5053 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-tmp-dir\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:57.137487 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:57.137488    5053 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-home\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:57.137667 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:57.137502    5053 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-model-cache\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:57.138827 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:57.138802    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-tls-certs" pod "9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9" (UID: "9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:15:57.138940 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:57.138844    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-kube-api-access-9lxxq" pod "9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9" (UID: "9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9"). InnerVolumeSpecName "kube-api-access-9lxxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:15:57.138940 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:57.138892    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-dshm" pod "9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9" (UID: "9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:15:57.239467 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:57.239434    5053 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-dshm\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:57.239467 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:57.239462    5053 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-tls-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:57.239467 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:57.239472    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lxxq\" (UniqueName: \"kubernetes.io/projected/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9-kube-api-access-9lxxq\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:57.943977 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:57.943937    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr" podUID="9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.234:8000/ready\": context deadline exceeded"
Nov 03 22:15:58.017700 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:58.017667    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"
Nov 03 22:15:58.017867 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:58.017701    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr" event={"ID":"9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9","Type":"ContainerDied","Data":"3854ef6422dbafbcdf2efbcbee60897c6aa708b31bb111f7d79f6a066e8f6e1d"}
Nov 03 22:15:58.017867 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:58.017766    5053 scope.go:122] "RemoveContainer" containerID="239708540b812c4ef9cc3a211b873d04327cdc43a273a344002f92710003f619"
Nov 03 22:15:58.183759 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:58.183711    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"]
Nov 03 22:15:58.186961 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:58.186938    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-75f577/auth-test-model-75f577-kserve-5cff888bbf-sqpnr"]
Nov 03 22:15:59.668126 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:59.668088    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9" path="/var/lib/kubelet/pods/9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9/volumes"
Nov 03 22:15:59.975711 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:59.975679    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"]
Nov 03 22:15:59.975909 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:15:59.975888    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4" podUID="6862760e-4710-4dc1-afd1-30df93440970" containerName="istio-proxy" containerID="cri-o://3d0794c985e0d28a7bdeed4f07be41eeccdd896e93007e7f77deede303965ade" gracePeriod=30
Nov 03 22:16:03.561847 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:03.561813    5053 patch_prober.go:29] interesting pod/e2e-autores-12dee4-openshift-default-577996447c-jqml4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:16:03.562339 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:03.561879    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4" podUID="6862760e-4710-4dc1-afd1-30df93440970" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:16:05.265059 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.265027    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:16:05.351451 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.351414    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-workload-socket\") pod \"6862760e-4710-4dc1-afd1-30df93440970\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") "
Nov 03 22:16:05.351651 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.351514    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6862760e-4710-4dc1-afd1-30df93440970-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6862760e-4710-4dc1-afd1-30df93440970-istio-podinfo\") pod \"6862760e-4710-4dc1-afd1-30df93440970\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") "
Nov 03 22:16:05.351651 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.351541    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-istio-data\") pod \"6862760e-4710-4dc1-afd1-30df93440970\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") "
Nov 03 22:16:05.351783 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.351690    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6862760e-4710-4dc1-afd1-30df93440970-istio-token\" (UniqueName: \"kubernetes.io/projected/6862760e-4710-4dc1-afd1-30df93440970-istio-token\") pod \"6862760e-4710-4dc1-afd1-30df93440970\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") "
Nov 03 22:16:05.351783 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.351712    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-workload-socket" pod "6862760e-4710-4dc1-afd1-30df93440970" (UID: "6862760e-4710-4dc1-afd1-30df93440970"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:16:05.351783 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.351744    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6862760e-4710-4dc1-afd1-30df93440970-kube-api-access-p8pb9\" (UniqueName: \"kubernetes.io/projected/6862760e-4710-4dc1-afd1-30df93440970-kube-api-access-p8pb9\") pod \"6862760e-4710-4dc1-afd1-30df93440970\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") "
Nov 03 22:16:05.351951 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.351828    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-credential-socket\") pod \"6862760e-4710-4dc1-afd1-30df93440970\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") "
Nov 03 22:16:05.351951 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.351856    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-workload-certs\") pod \"6862760e-4710-4dc1-afd1-30df93440970\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") "
Nov 03 22:16:05.351951 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.351904    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-istio-envoy\") pod \"6862760e-4710-4dc1-afd1-30df93440970\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") "
Nov 03 22:16:05.352099 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.351948    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6862760e-4710-4dc1-afd1-30df93440970-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6862760e-4710-4dc1-afd1-30df93440970-istiod-ca-cert\") pod \"6862760e-4710-4dc1-afd1-30df93440970\" (UID: \"6862760e-4710-4dc1-afd1-30df93440970\") "
Nov 03 22:16:05.352099 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.351856    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-istio-data" pod "6862760e-4710-4dc1-afd1-30df93440970" (UID: "6862760e-4710-4dc1-afd1-30df93440970"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:16:05.352099 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.352024    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-credential-socket" pod "6862760e-4710-4dc1-afd1-30df93440970" (UID: "6862760e-4710-4dc1-afd1-30df93440970"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:16:05.352354 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.352304    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-workload-certs" pod "6862760e-4710-4dc1-afd1-30df93440970" (UID: "6862760e-4710-4dc1-afd1-30df93440970"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:16:05.352354 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.352340    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6862760e-4710-4dc1-afd1-30df93440970-istiod-ca-cert" pod "6862760e-4710-4dc1-afd1-30df93440970" (UID: "6862760e-4710-4dc1-afd1-30df93440970"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:16:05.352756 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.352736    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:16:05.352805 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.352764    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:16:05.352805 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.352780    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:16:05.352805 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.352794    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:16:05.352972 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.352808    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6862760e-4710-4dc1-afd1-30df93440970-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:16:05.354127 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.354102    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6862760e-4710-4dc1-afd1-30df93440970-istio-token" pod "6862760e-4710-4dc1-afd1-30df93440970" (UID: "6862760e-4710-4dc1-afd1-30df93440970"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:16:05.354279 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.354211    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-istio-envoy" pod "6862760e-4710-4dc1-afd1-30df93440970" (UID: "6862760e-4710-4dc1-afd1-30df93440970"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:16:05.354279 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.354228    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6862760e-4710-4dc1-afd1-30df93440970-istio-podinfo" pod "6862760e-4710-4dc1-afd1-30df93440970" (UID: "6862760e-4710-4dc1-afd1-30df93440970"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:16:05.354368 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.354303    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6862760e-4710-4dc1-afd1-30df93440970-kube-api-access-p8pb9" pod "6862760e-4710-4dc1-afd1-30df93440970" (UID: "6862760e-4710-4dc1-afd1-30df93440970"). InnerVolumeSpecName "kube-api-access-p8pb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:16:05.453853 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.453816    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6862760e-4710-4dc1-afd1-30df93440970-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:16:05.453853 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.453849    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8pb9\" (UniqueName: \"kubernetes.io/projected/6862760e-4710-4dc1-afd1-30df93440970-kube-api-access-p8pb9\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:16:05.453853 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.453860    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6862760e-4710-4dc1-afd1-30df93440970-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:16:05.454083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:05.453868    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6862760e-4710-4dc1-afd1-30df93440970-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:16:06.079772 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:06.079737    5053 generic.go:356] "Generic (PLEG): container finished" podID="6862760e-4710-4dc1-afd1-30df93440970" containerID="3d0794c985e0d28a7bdeed4f07be41eeccdd896e93007e7f77deede303965ade" exitCode=0
Nov 03 22:16:06.079971 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:06.079801    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"
Nov 03 22:16:06.079971 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:06.079820    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4" event={"ID":"6862760e-4710-4dc1-afd1-30df93440970","Type":"ContainerDied","Data":"3d0794c985e0d28a7bdeed4f07be41eeccdd896e93007e7f77deede303965ade"}
Nov 03 22:16:06.079971 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:06.079876    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4" event={"ID":"6862760e-4710-4dc1-afd1-30df93440970","Type":"ContainerDied","Data":"1f84b8881584637138181e0626b05ceb0611854197c9c14c7206b2e9cff4ad36"}
Nov 03 22:16:06.079971 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:06.079898    5053 scope.go:122] "RemoveContainer" containerID="3d0794c985e0d28a7bdeed4f07be41eeccdd896e93007e7f77deede303965ade"
Nov 03 22:16:06.090195 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:06.090175    5053 scope.go:122] "RemoveContainer" containerID="7e251468046f1aa4bec1d2389207a5003941723afbc5f1f63f051afecf56d636"
Nov 03 22:16:06.099766 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:06.099578    5053 scope.go:122] "RemoveContainer" containerID="3d0794c985e0d28a7bdeed4f07be41eeccdd896e93007e7f77deede303965ade"
Nov 03 22:16:06.099882 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:16:06.099856    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0794c985e0d28a7bdeed4f07be41eeccdd896e93007e7f77deede303965ade\": container with ID starting with 3d0794c985e0d28a7bdeed4f07be41eeccdd896e93007e7f77deede303965ade not found: ID does not exist" containerID="3d0794c985e0d28a7bdeed4f07be41eeccdd896e93007e7f77deede303965ade"
Nov 03 22:16:06.099934 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:06.099881    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0794c985e0d28a7bdeed4f07be41eeccdd896e93007e7f77deede303965ade"} err="failed to get container status \"3d0794c985e0d28a7bdeed4f07be41eeccdd896e93007e7f77deede303965ade\": rpc error: code = NotFound desc = could not find container \"3d0794c985e0d28a7bdeed4f07be41eeccdd896e93007e7f77deede303965ade\": container with ID starting with 3d0794c985e0d28a7bdeed4f07be41eeccdd896e93007e7f77deede303965ade not found: ID does not exist"
Nov 03 22:16:06.099934 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:06.099898    5053 scope.go:122] "RemoveContainer" containerID="7e251468046f1aa4bec1d2389207a5003941723afbc5f1f63f051afecf56d636"
Nov 03 22:16:06.100148 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:16:06.100128    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e251468046f1aa4bec1d2389207a5003941723afbc5f1f63f051afecf56d636\": container with ID starting with 7e251468046f1aa4bec1d2389207a5003941723afbc5f1f63f051afecf56d636 not found: ID does not exist" containerID="7e251468046f1aa4bec1d2389207a5003941723afbc5f1f63f051afecf56d636"
Nov 03 22:16:06.100205 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:06.100157    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e251468046f1aa4bec1d2389207a5003941723afbc5f1f63f051afecf56d636"} err="failed to get container status \"7e251468046f1aa4bec1d2389207a5003941723afbc5f1f63f051afecf56d636\": rpc error: code = NotFound desc = could not find container \"7e251468046f1aa4bec1d2389207a5003941723afbc5f1f63f051afecf56d636\": container with ID starting with 7e251468046f1aa4bec1d2389207a5003941723afbc5f1f63f051afecf56d636 not found: ID does not exist"
Nov 03 22:16:06.296960 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:06.296928    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"]
Nov 03 22:16:06.346752 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:06.346663    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-12dee4-openshift-default-577996447c-jqml4"]
Nov 03 22:16:07.667485 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:07.667457    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6862760e-4710-4dc1-afd1-30df93440970" path="/var/lib/kubelet/pods/6862760e-4710-4dc1-afd1-30df93440970/volumes"
Nov 03 22:16:08.442289 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.442258    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"]
Nov 03 22:16:08.442960 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.442934    5053 cpu_manager.go:414] "containerMap: removing container" podUID="a83ccbc3-6590-43cd-bda1-1e8f478060c1" containerName="cleanup"
Nov 03 22:16:08.443114 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.442960    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83ccbc3-6590-43cd-bda1-1e8f478060c1" containerName="cleanup"
Nov 03 22:16:08.443114 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.443018    5053 cpu_manager.go:414] "containerMap: removing container" podUID="6862760e-4710-4dc1-afd1-30df93440970" containerName="istio-proxy"
Nov 03 22:16:08.443114 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.443027    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6862760e-4710-4dc1-afd1-30df93440970" containerName="istio-proxy"
Nov 03 22:16:08.443114 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.443040    5053 cpu_manager.go:414] "containerMap: removing container" podUID="a83ccbc3-6590-43cd-bda1-1e8f478060c1" containerName="cleanup"
Nov 03 22:16:08.443114 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.443046    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83ccbc3-6590-43cd-bda1-1e8f478060c1" containerName="cleanup"
Nov 03 22:16:08.443114 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.443086    5053 cpu_manager.go:414] "containerMap: removing container" podUID="6862760e-4710-4dc1-afd1-30df93440970" containerName="istio-proxy"
Nov 03 22:16:08.443114 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.443094    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6862760e-4710-4dc1-afd1-30df93440970" containerName="istio-proxy"
Nov 03 22:16:08.443114 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.443107    5053 cpu_manager.go:414] "containerMap: removing container" podUID="f5db8aaf-b362-4a27-b828-54655dff7ad1" containerName="cleanup"
Nov 03 22:16:08.443114 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.443116    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5db8aaf-b362-4a27-b828-54655dff7ad1" containerName="cleanup"
Nov 03 22:16:08.443560 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.443161    5053 cpu_manager.go:414] "containerMap: removing container" podUID="94f8d07a-8d67-4366-97b2-2773315a8ec0" containerName="cleanup"
Nov 03 22:16:08.443560 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.443168    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f8d07a-8d67-4366-97b2-2773315a8ec0" containerName="cleanup"
Nov 03 22:16:08.443560 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.443177    5053 cpu_manager.go:414] "containerMap: removing container" podUID="762459e3-f8a0-40f3-bd1c-24eaa187d79a" containerName="revoke-keys"
Nov 03 22:16:08.443560 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.443183    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="762459e3-f8a0-40f3-bd1c-24eaa187d79a" containerName="revoke-keys"
Nov 03 22:16:08.443560 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.443199    5053 cpu_manager.go:414] "containerMap: removing container" podUID="9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9" containerName="main"
Nov 03 22:16:08.443560 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.443206    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9" containerName="main"
Nov 03 22:16:08.443560 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.443220    5053 cpu_manager.go:414] "containerMap: removing container" podUID="79ff394d-d937-46b1-b994-e180a924a3e0" containerName="maas-api"
Nov 03 22:16:08.443560 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.443226    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ff394d-d937-46b1-b994-e180a924a3e0" containerName="maas-api"
Nov 03 22:16:08.443560 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.443407    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="762459e3-f8a0-40f3-bd1c-24eaa187d79a" containerName="revoke-keys"
Nov 03 22:16:08.443560 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.443420    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5db8aaf-b362-4a27-b828-54655dff7ad1" containerName="cleanup"
Nov 03 22:16:08.443560 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.443430    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f6f96cd-70e3-445b-8ccf-9d305d5f6ee9" containerName="main"
Nov 03 22:16:08.443560 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.443438    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ff394d-d937-46b1-b994-e180a924a3e0" containerName="maas-api"
Nov 03 22:16:08.443560 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.443450    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83ccbc3-6590-43cd-bda1-1e8f478060c1" containerName="cleanup"
Nov 03 22:16:08.443560 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.443457    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="6862760e-4710-4dc1-afd1-30df93440970" containerName="istio-proxy"
Nov 03 22:16:08.443560 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.443465    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83ccbc3-6590-43cd-bda1-1e8f478060c1" containerName="cleanup"
Nov 03 22:16:08.448442 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.448423    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.500680 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.500648    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-f3e839-openshift-default-dockercfg-mmmw5\""
Nov 03 22:16:08.554259 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.554208    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"]
Nov 03 22:16:08.595132 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.595090    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-data\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.595432 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.595159    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-envoy\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.595432 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.595303    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-podinfo\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.595432 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.595395    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d8d06b2-a794-4262-855d-75c8e268f3b5-istiod-ca-cert\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.595609 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.595490    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-workload-socket\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.595609 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.595574    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-workload-certs\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.595703 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.595619    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-credential-socket\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.595786 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.595760    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-token\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.595927 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.595823    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrx2l\" (UniqueName: \"kubernetes.io/projected/3d8d06b2-a794-4262-855d-75c8e268f3b5-kube-api-access-lrx2l\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.701291 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.701252    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-data\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.701941 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.701913    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-envoy\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.702105 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.702088    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-podinfo\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.711709 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.711665    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d8d06b2-a794-4262-855d-75c8e268f3b5-istiod-ca-cert\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.711947 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.711925    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-workload-socket\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.737261 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.733338    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-workload-certs\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.737261 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.733464    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-credential-socket\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.737261 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.733589    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-token\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.737261 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.733659    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrx2l\" (UniqueName: \"kubernetes.io/projected/3d8d06b2-a794-4262-855d-75c8e268f3b5-kube-api-access-lrx2l\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.737261 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.734617    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-credential-socket\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.737261 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.736647    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-workload-socket\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.737261 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.710668    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-data\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.737784 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.737172    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d8d06b2-a794-4262-855d-75c8e268f3b5-istiod-ca-cert\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.738763 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.738733    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-workload-certs\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.767344 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.766948    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-envoy\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.774920 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.774886    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-podinfo\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.795370 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.795345    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-token\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:08.797465 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:08.797144    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrx2l\" (UniqueName: \"kubernetes.io/projected/3d8d06b2-a794-4262-855d-75c8e268f3b5-kube-api-access-lrx2l\") pod \"e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") " pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:09.062948 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:09.062859    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:09.493960 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:09.493931    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"]
Nov 03 22:16:09.534288 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:16:09.534261    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8d06b2_a794_4262_855d_75c8e268f3b5.slice/crio-f7e6dbceaa8ca0ff29366ff48c74a678a0ac8fd755531187609d0fd284134ed2 WatchSource:0}: Error finding container f7e6dbceaa8ca0ff29366ff48c74a678a0ac8fd755531187609d0fd284134ed2: Status 404 returned error can't find the container with id f7e6dbceaa8ca0ff29366ff48c74a678a0ac8fd755531187609d0fd284134ed2
Nov 03 22:16:09.536879 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:09.536842    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:16:09.536986 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:09.536916    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:16:09.536986 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:09.536945    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:16:10.111000 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:10.110962    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29" event={"ID":"3d8d06b2-a794-4262-855d-75c8e268f3b5","Type":"ContainerStarted","Data":"03b5ef7c336998e491f6a28e6c89392f47ee3acf52e32c09168151623d0485bd"}
Nov 03 22:16:10.111478 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:10.111022    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29" event={"ID":"3d8d06b2-a794-4262-855d-75c8e268f3b5","Type":"ContainerStarted","Data":"f7e6dbceaa8ca0ff29366ff48c74a678a0ac8fd755531187609d0fd284134ed2"}
Nov 03 22:16:10.929153 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:16:10.929113    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:16:11.063041 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:11.063004    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:11.070286 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:11.070258    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:11.121570 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:11.121540    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:11.123025 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:11.122992    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:16:11.237031 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:11.236970    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29" podStartSLOduration=3.236953385 podStartE2EDuration="3.236953385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:16:10.325597628 +0000 UTC m=+3684.819519503" watchObservedRunningTime="2026-11-03 22:16:11.236953385 +0000 UTC m=+3685.730875322"
Nov 03 22:16:22.516679 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:22.516636    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"]
Nov 03 22:16:22.517189 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:22.516869    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9" podUID="b5a54168-03bd-4b47-9c62-a9de6168027b" containerName="istio-proxy" containerID="cri-o://1f7152660eeb2fa9cdaa13784b3cab4d96a6d3eb4f23418168d58c2f9382b994" gracePeriod=30
Nov 03 22:16:23.251352 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:23.251305    5053 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:16:23.251533 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:23.251378    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9" podUID="b5a54168-03bd-4b47-9c62-a9de6168027b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:16:27.841934 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:27.841900    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 22:16:27.902530 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:27.902495    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-podinfo\") pod \"b5a54168-03bd-4b47-9c62-a9de6168027b\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") "
Nov 03 22:16:27.902741 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:27.902596    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-token\" (UniqueName: \"kubernetes.io/projected/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-token\") pod \"b5a54168-03bd-4b47-9c62-a9de6168027b\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") "
Nov 03 22:16:27.902741 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:27.902642    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-credential-socket\") pod \"b5a54168-03bd-4b47-9c62-a9de6168027b\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") "
Nov 03 22:16:27.902741 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:27.902670    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-envoy\") pod \"b5a54168-03bd-4b47-9c62-a9de6168027b\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") "
Nov 03 22:16:27.902741 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:27.902695    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-data\") pod \"b5a54168-03bd-4b47-9c62-a9de6168027b\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") "
Nov 03 22:16:27.902966 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:27.902739    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a54168-03bd-4b47-9c62-a9de6168027b-kube-api-access-hf62w\" (UniqueName: \"kubernetes.io/projected/b5a54168-03bd-4b47-9c62-a9de6168027b-kube-api-access-hf62w\") pod \"b5a54168-03bd-4b47-9c62-a9de6168027b\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") "
Nov 03 22:16:27.902966 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:27.902802    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-workload-certs\") pod \"b5a54168-03bd-4b47-9c62-a9de6168027b\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") "
Nov 03 22:16:27.902966 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:27.902869    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a54168-03bd-4b47-9c62-a9de6168027b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a54168-03bd-4b47-9c62-a9de6168027b-istiod-ca-cert\") pod \"b5a54168-03bd-4b47-9c62-a9de6168027b\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") "
Nov 03 22:16:27.902966 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:27.902897    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-workload-socket\") pod \"b5a54168-03bd-4b47-9c62-a9de6168027b\" (UID: \"b5a54168-03bd-4b47-9c62-a9de6168027b\") "
Nov 03 22:16:27.903199 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:27.903061    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-credential-socket" pod "b5a54168-03bd-4b47-9c62-a9de6168027b" (UID: "b5a54168-03bd-4b47-9c62-a9de6168027b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:16:27.903274 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:27.903252    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-workload-socket" pod "b5a54168-03bd-4b47-9c62-a9de6168027b" (UID: "b5a54168-03bd-4b47-9c62-a9de6168027b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:16:27.903690 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:27.903458    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-workload-certs" pod "b5a54168-03bd-4b47-9c62-a9de6168027b" (UID: "b5a54168-03bd-4b47-9c62-a9de6168027b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:16:27.903690 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:27.903537    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a54168-03bd-4b47-9c62-a9de6168027b-istiod-ca-cert" pod "b5a54168-03bd-4b47-9c62-a9de6168027b" (UID: "b5a54168-03bd-4b47-9c62-a9de6168027b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:16:27.903690 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:27.903544    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:16:27.903690 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:27.903561    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:16:27.903690 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:27.903651    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-data" pod "b5a54168-03bd-4b47-9c62-a9de6168027b" (UID: "b5a54168-03bd-4b47-9c62-a9de6168027b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:16:27.905641 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:27.905609    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-podinfo" pod "b5a54168-03bd-4b47-9c62-a9de6168027b" (UID: "b5a54168-03bd-4b47-9c62-a9de6168027b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:16:27.905767 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:27.905636    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a54168-03bd-4b47-9c62-a9de6168027b-kube-api-access-hf62w" pod "b5a54168-03bd-4b47-9c62-a9de6168027b" (UID: "b5a54168-03bd-4b47-9c62-a9de6168027b"). InnerVolumeSpecName "kube-api-access-hf62w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:16:27.905767 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:27.905652    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-token" pod "b5a54168-03bd-4b47-9c62-a9de6168027b" (UID: "b5a54168-03bd-4b47-9c62-a9de6168027b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:16:27.905767 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:27.905728    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-envoy" pod "b5a54168-03bd-4b47-9c62-a9de6168027b" (UID: "b5a54168-03bd-4b47-9c62-a9de6168027b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:16:28.004667 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:28.004624    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:16:28.004667 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:28.004663    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:16:28.004667 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:28.004674    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf62w\" (UniqueName: \"kubernetes.io/projected/b5a54168-03bd-4b47-9c62-a9de6168027b-kube-api-access-hf62w\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:16:28.004935 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:28.004685    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a54168-03bd-4b47-9c62-a9de6168027b-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:16:28.004935 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:28.004694    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a54168-03bd-4b47-9c62-a9de6168027b-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:16:28.004935 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:28.004702    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:16:28.004935 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:28.004710    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a54168-03bd-4b47-9c62-a9de6168027b-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:16:28.239654 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:28.239613    5053 generic.go:356] "Generic (PLEG): container finished" podID="b5a54168-03bd-4b47-9c62-a9de6168027b" containerID="1f7152660eeb2fa9cdaa13784b3cab4d96a6d3eb4f23418168d58c2f9382b994" exitCode=0
Nov 03 22:16:28.239890 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:28.239670    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"
Nov 03 22:16:28.239890 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:28.239730    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9" event={"ID":"b5a54168-03bd-4b47-9c62-a9de6168027b","Type":"ContainerDied","Data":"1f7152660eeb2fa9cdaa13784b3cab4d96a6d3eb4f23418168d58c2f9382b994"}
Nov 03 22:16:28.239890 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:28.239773    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9" event={"ID":"b5a54168-03bd-4b47-9c62-a9de6168027b","Type":"ContainerDied","Data":"153ceb77324c12ba17131cfb9624e034a4553f80a94ff4ca5ae178f35b6b3300"}
Nov 03 22:16:28.239890 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:28.239799    5053 scope.go:122] "RemoveContainer" containerID="1f7152660eeb2fa9cdaa13784b3cab4d96a6d3eb4f23418168d58c2f9382b994"
Nov 03 22:16:28.252903 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:28.252871    5053 scope.go:122] "RemoveContainer" containerID="1f7152660eeb2fa9cdaa13784b3cab4d96a6d3eb4f23418168d58c2f9382b994"
Nov 03 22:16:28.253269 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:16:28.253227    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f7152660eeb2fa9cdaa13784b3cab4d96a6d3eb4f23418168d58c2f9382b994\": container with ID starting with 1f7152660eeb2fa9cdaa13784b3cab4d96a6d3eb4f23418168d58c2f9382b994 not found: ID does not exist" containerID="1f7152660eeb2fa9cdaa13784b3cab4d96a6d3eb4f23418168d58c2f9382b994"
Nov 03 22:16:28.253372 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:28.253281    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7152660eeb2fa9cdaa13784b3cab4d96a6d3eb4f23418168d58c2f9382b994"} err="failed to get container status \"1f7152660eeb2fa9cdaa13784b3cab4d96a6d3eb4f23418168d58c2f9382b994\": rpc error: code = NotFound desc = could not find container \"1f7152660eeb2fa9cdaa13784b3cab4d96a6d3eb4f23418168d58c2f9382b994\": container with ID starting with 1f7152660eeb2fa9cdaa13784b3cab4d96a6d3eb4f23418168d58c2f9382b994 not found: ID does not exist"
Nov 03 22:16:28.417668 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:28.417629    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"]
Nov 03 22:16:28.428861 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:28.428833    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-hklf9"]
Nov 03 22:16:29.667604 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:29.667574    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5a54168-03bd-4b47-9c62-a9de6168027b" path="/var/lib/kubelet/pods/b5a54168-03bd-4b47-9c62-a9de6168027b/volumes"
Nov 03 22:16:37.719253 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:37.719159    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"]
Nov 03 22:16:37.719865 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:37.719849    5053 cpu_manager.go:414] "containerMap: removing container" podUID="a83ccbc3-6590-43cd-bda1-1e8f478060c1" containerName="cleanup"
Nov 03 22:16:37.719912 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:37.719867    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83ccbc3-6590-43cd-bda1-1e8f478060c1" containerName="cleanup"
Nov 03 22:16:37.719912 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:37.719898    5053 cpu_manager.go:414] "containerMap: removing container" podUID="b5a54168-03bd-4b47-9c62-a9de6168027b" containerName="istio-proxy"
Nov 03 22:16:37.719912 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:37.719903    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a54168-03bd-4b47-9c62-a9de6168027b" containerName="istio-proxy"
Nov 03 22:16:37.720012 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:37.719983    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a54168-03bd-4b47-9c62-a9de6168027b" containerName="istio-proxy"
Nov 03 22:16:37.720012 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:37.719999    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="6862760e-4710-4dc1-afd1-30df93440970" containerName="istio-proxy"
Nov 03 22:16:37.720012 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:37.720007    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83ccbc3-6590-43cd-bda1-1e8f478060c1" containerName="cleanup"
Nov 03 22:16:37.724582 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:37.724559    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"
Nov 03 22:16:37.779854 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:37.779818    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"]
Nov 03 22:16:37.825166 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:37.825130    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-534c77-serving-cert\""
Nov 03 22:16:37.840470 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:37.840429    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frl6x\" (UniqueName: \"kubernetes.io/projected/9cbef392-ef4f-475b-a326-d3aaf057d256-kube-api-access-frl6x\") pod \"maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh\" (UID: \"9cbef392-ef4f-475b-a326-d3aaf057d256\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"
Nov 03 22:16:37.840857 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:37.840836    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cbef392-ef4f-475b-a326-d3aaf057d256-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh\" (UID: \"9cbef392-ef4f-475b-a326-d3aaf057d256\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"
Nov 03 22:16:37.841017 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:37.841000    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cbef392-ef4f-475b-a326-d3aaf057d256-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh\" (UID: \"9cbef392-ef4f-475b-a326-d3aaf057d256\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"
Nov 03 22:16:37.841069 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:37.841025    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cbef392-ef4f-475b-a326-d3aaf057d256-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh\" (UID: \"9cbef392-ef4f-475b-a326-d3aaf057d256\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"
Nov 03 22:16:37.942385 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:37.942351    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cbef392-ef4f-475b-a326-d3aaf057d256-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh\" (UID: \"9cbef392-ef4f-475b-a326-d3aaf057d256\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"
Nov 03 22:16:37.942597 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:37.942470    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cbef392-ef4f-475b-a326-d3aaf057d256-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh\" (UID: \"9cbef392-ef4f-475b-a326-d3aaf057d256\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"
Nov 03 22:16:37.942597 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:37.942495    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cbef392-ef4f-475b-a326-d3aaf057d256-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh\" (UID: \"9cbef392-ef4f-475b-a326-d3aaf057d256\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"
Nov 03 22:16:37.942597 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:16:37.942597    5053 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-serving-cert: secret "maas-api-e2e-shared-b-w1-534c77-serving-cert" not found
Nov 03 22:16:37.942783 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:16:37.942677    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cbef392-ef4f-475b-a326-d3aaf057d256-maas-api-tls podName:9cbef392-ef4f-475b-a326-d3aaf057d256 nodeName:}" failed. No retries permitted until 2026-11-03 22:16:38.442655067 +0000 UTC m=+3712.936576945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9cbef392-ef4f-475b-a326-d3aaf057d256-maas-api-tls") pod "maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh" (UID: "9cbef392-ef4f-475b-a326-d3aaf057d256") : secret "maas-api-e2e-shared-b-w1-534c77-serving-cert" not found
Nov 03 22:16:37.943229 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:37.943199    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cbef392-ef4f-475b-a326-d3aaf057d256-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh\" (UID: \"9cbef392-ef4f-475b-a326-d3aaf057d256\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"
Nov 03 22:16:37.943430 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:37.943408    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frl6x\" (UniqueName: \"kubernetes.io/projected/9cbef392-ef4f-475b-a326-d3aaf057d256-kube-api-access-frl6x\") pod \"maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh\" (UID: \"9cbef392-ef4f-475b-a326-d3aaf057d256\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"
Nov 03 22:16:37.946535 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:37.946486    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cbef392-ef4f-475b-a326-d3aaf057d256-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh\" (UID: \"9cbef392-ef4f-475b-a326-d3aaf057d256\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"
Nov 03 22:16:38.001090 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:38.001012    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frl6x\" (UniqueName: \"kubernetes.io/projected/9cbef392-ef4f-475b-a326-d3aaf057d256-kube-api-access-frl6x\") pod \"maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh\" (UID: \"9cbef392-ef4f-475b-a326-d3aaf057d256\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"
Nov 03 22:16:38.450854 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:38.450815    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cbef392-ef4f-475b-a326-d3aaf057d256-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh\" (UID: \"9cbef392-ef4f-475b-a326-d3aaf057d256\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"
Nov 03 22:16:38.453714 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:38.453684    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cbef392-ef4f-475b-a326-d3aaf057d256-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh\" (UID: \"9cbef392-ef4f-475b-a326-d3aaf057d256\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"
Nov 03 22:16:38.639422 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:38.639390    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"
Nov 03 22:16:39.018509 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:39.018483    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"]
Nov 03 22:16:39.019873 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:16:39.019845    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cbef392_ef4f_475b_a326_d3aaf057d256.slice/crio-233f49a04d16a0699ae9ed12f73aecf8ef2a381b3ca6ba03db78ec2e537f3c95 WatchSource:0}: Error finding container 233f49a04d16a0699ae9ed12f73aecf8ef2a381b3ca6ba03db78ec2e537f3c95: Status 404 returned error can't find the container with id 233f49a04d16a0699ae9ed12f73aecf8ef2a381b3ca6ba03db78ec2e537f3c95
Nov 03 22:16:39.319578 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:39.319496    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh" event={"ID":"9cbef392-ef4f-475b-a326-d3aaf057d256","Type":"ContainerStarted","Data":"233f49a04d16a0699ae9ed12f73aecf8ef2a381b3ca6ba03db78ec2e537f3c95"}
Nov 03 22:16:40.328750 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:40.328701    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh" event={"ID":"9cbef392-ef4f-475b-a326-d3aaf057d256","Type":"ContainerStarted","Data":"c8b0df52bec098d47ac04e7d21d96554c6770b69f68e74485eaae9e090bdeaf0"}
Nov 03 22:16:40.329172 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:40.328807    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"
Nov 03 22:16:40.932373 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:16:40.932337    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:16:46.344002 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:46.343971    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"
Nov 03 22:16:46.457123 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:16:46.457076    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh" podStartSLOduration=8.643572201 podStartE2EDuration="9.45706095s" totalImagesPullingTime="813.488749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:16:40.442467821 +0000 UTC m=+3714.936389708" watchObservedRunningTime="2026-11-03 22:16:46.45706095 +0000 UTC m=+3720.950982807"
Nov 03 22:17:06.671209 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:06.671169    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn"]
Nov 03 22:17:06.675796 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:06.675776    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn"
Nov 03 22:17:06.729832 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:06.729795    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-d7b12e-serving-cert\""
Nov 03 22:17:06.738520 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:06.738484    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn"]
Nov 03 22:17:06.802370 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:06.802323    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/722a009c-bbee-44ea-b624-bd8eeb0897b8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn\" (UID: \"722a009c-bbee-44ea-b624-bd8eeb0897b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn"
Nov 03 22:17:06.802866 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:06.802837    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/722a009c-bbee-44ea-b624-bd8eeb0897b8-metrics-certs\") pod \"maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn\" (UID: \"722a009c-bbee-44ea-b624-bd8eeb0897b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn"
Nov 03 22:17:06.803105 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:06.803087    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/722a009c-bbee-44ea-b624-bd8eeb0897b8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn\" (UID: \"722a009c-bbee-44ea-b624-bd8eeb0897b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn"
Nov 03 22:17:06.803376 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:06.803354    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbplp\" (UniqueName: \"kubernetes.io/projected/722a009c-bbee-44ea-b624-bd8eeb0897b8-kube-api-access-wbplp\") pod \"maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn\" (UID: \"722a009c-bbee-44ea-b624-bd8eeb0897b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn"
Nov 03 22:17:06.905549 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:06.905507    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/722a009c-bbee-44ea-b624-bd8eeb0897b8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn\" (UID: \"722a009c-bbee-44ea-b624-bd8eeb0897b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn"
Nov 03 22:17:06.905721 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:06.905586    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/722a009c-bbee-44ea-b624-bd8eeb0897b8-metrics-certs\") pod \"maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn\" (UID: \"722a009c-bbee-44ea-b624-bd8eeb0897b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn"
Nov 03 22:17:06.905721 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:06.905645    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/722a009c-bbee-44ea-b624-bd8eeb0897b8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn\" (UID: \"722a009c-bbee-44ea-b624-bd8eeb0897b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn"
Nov 03 22:17:06.905721 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:06.905686    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbplp\" (UniqueName: \"kubernetes.io/projected/722a009c-bbee-44ea-b624-bd8eeb0897b8-kube-api-access-wbplp\") pod \"maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn\" (UID: \"722a009c-bbee-44ea-b624-bd8eeb0897b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn"
Nov 03 22:17:06.907068 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:06.907044    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/722a009c-bbee-44ea-b624-bd8eeb0897b8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn\" (UID: \"722a009c-bbee-44ea-b624-bd8eeb0897b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn"
Nov 03 22:17:06.910461 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:06.910435    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/722a009c-bbee-44ea-b624-bd8eeb0897b8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn\" (UID: \"722a009c-bbee-44ea-b624-bd8eeb0897b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn"
Nov 03 22:17:06.911685 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:06.910722    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/722a009c-bbee-44ea-b624-bd8eeb0897b8-metrics-certs\") pod \"maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn\" (UID: \"722a009c-bbee-44ea-b624-bd8eeb0897b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn"
Nov 03 22:17:06.965875 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:06.965841    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbplp\" (UniqueName: \"kubernetes.io/projected/722a009c-bbee-44ea-b624-bd8eeb0897b8-kube-api-access-wbplp\") pod \"maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn\" (UID: \"722a009c-bbee-44ea-b624-bd8eeb0897b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn"
Nov 03 22:17:07.001741 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:07.001702    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn"
Nov 03 22:17:07.400806 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:07.400762    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn"]
Nov 03 22:17:07.402121 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:17:07.402074    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722a009c_bbee_44ea_b624_bd8eeb0897b8.slice/crio-98ca7d4621c6f2ee73aeade7fcb1700708a54641996793d5dd2b188c33bce9e1 WatchSource:0}: Error finding container 98ca7d4621c6f2ee73aeade7fcb1700708a54641996793d5dd2b188c33bce9e1: Status 404 returned error can't find the container with id 98ca7d4621c6f2ee73aeade7fcb1700708a54641996793d5dd2b188c33bce9e1
Nov 03 22:17:07.403730 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:07.403698    5053 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 22:17:07.541448 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:07.541409    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn" event={"ID":"722a009c-bbee-44ea-b624-bd8eeb0897b8","Type":"ContainerStarted","Data":"98ca7d4621c6f2ee73aeade7fcb1700708a54641996793d5dd2b188c33bce9e1"}
Nov 03 22:17:08.552805 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:08.552758    5053 generic.go:356] "Generic (PLEG): container finished" podID="886c2db8-998e-4425-a064-630afe2da8ca" containerID="81b9d9965618291f44b9f9b6b9b57a75adf2913ca0cfd05380f66caaed42325d" exitCode=0
Nov 03 22:17:08.553377 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:08.552835    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689" event={"ID":"886c2db8-998e-4425-a064-630afe2da8ca","Type":"ContainerDied","Data":"81b9d9965618291f44b9f9b6b9b57a75adf2913ca0cfd05380f66caaed42325d"}
Nov 03 22:17:08.553465 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:08.553393    5053 scope.go:122] "RemoveContainer" containerID="81b9d9965618291f44b9f9b6b9b57a75adf2913ca0cfd05380f66caaed42325d"
Nov 03 22:17:08.554806 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:08.554779    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn" event={"ID":"722a009c-bbee-44ea-b624-bd8eeb0897b8","Type":"ContainerStarted","Data":"72c177d59490bdba1e17001749183d94e3afe4a7c2d7a364492224d0801162ea"}
Nov 03 22:17:08.554957 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:08.554943    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn"
Nov 03 22:17:08.555164 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:17:08.555140    5053 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod886c2db8_998e_4425_a064_630afe2da8ca.slice/crio-conmon-81b9d9965618291f44b9f9b6b9b57a75adf2913ca0cfd05380f66caaed42325d.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 22:17:08.555327 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:08.555300    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:17:08.555417 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:08.555389    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:17:08.555494 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:08.555476    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:17:08.986892 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:08.986836    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn" podStartSLOduration=2.050694731 podStartE2EDuration="2.986820659s" totalImagesPullingTime="936.125928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:08.984055951 +0000 UTC m=+3743.477977821" watchObservedRunningTime="2026-11-03 22:17:08.986820659 +0000 UTC m=+3743.480742534"
Nov 03 22:17:09.564478 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:09.564443    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689" event={"ID":"886c2db8-998e-4425-a064-630afe2da8ca","Type":"ContainerStarted","Data":"2842c689efc31dc4c76aa032f86ae8aee5434257e7830eae950b80505b677411"}
Nov 03 22:17:10.464814 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:10.464766    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:17:10.465026 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:10.464989    5053 patch_prober.go:29] interesting pod/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 03 22:17:10.465078 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:10.465041    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689" podUID="886c2db8-998e-4425-a064-630afe2da8ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 03 22:17:10.956178 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:17:10.956141    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:17:11.464502 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:11.464468    5053 patch_prober.go:29] interesting pod/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 03 22:17:11.464708 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:11.464521    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689" podUID="886c2db8-998e-4425-a064-630afe2da8ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 03 22:17:12.464249 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:12.464210    5053 patch_prober.go:29] interesting pod/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 03 22:17:12.464755 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:12.464288    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689" podUID="886c2db8-998e-4425-a064-630afe2da8ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 03 22:17:13.465204 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:13.465164    5053 patch_prober.go:29] interesting pod/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 03 22:17:13.465678 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:13.465232    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689" podUID="886c2db8-998e-4425-a064-630afe2da8ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 03 22:17:14.464518 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:14.464484    5053 patch_prober.go:29] interesting pod/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 03 22:17:14.464711 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:14.464535    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689" podUID="886c2db8-998e-4425-a064-630afe2da8ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 03 22:17:14.570872 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:14.570832    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn"
Nov 03 22:17:15.469376 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:15.469342    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:17:15.469846 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:15.469819    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:17:15.470656 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:15.470637    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:17:39.999921 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:39.999875    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"]
Nov 03 22:17:40.004496 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:40.004478    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"
Nov 03 22:17:40.060493 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:40.060462    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"]
Nov 03 22:17:40.085563 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:40.085520    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/982a04b0-d1b7-4f74-ab18-f84ebc337092-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p\" (UID: \"982a04b0-d1b7-4f74-ab18-f84ebc337092\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"
Nov 03 22:17:40.085738 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:40.085664    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fmkv\" (UniqueName: \"kubernetes.io/projected/982a04b0-d1b7-4f74-ab18-f84ebc337092-kube-api-access-2fmkv\") pod \"maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p\" (UID: \"982a04b0-d1b7-4f74-ab18-f84ebc337092\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"
Nov 03 22:17:40.085738 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:40.085711    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/982a04b0-d1b7-4f74-ab18-f84ebc337092-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p\" (UID: \"982a04b0-d1b7-4f74-ab18-f84ebc337092\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"
Nov 03 22:17:40.085860 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:40.085738    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982a04b0-d1b7-4f74-ab18-f84ebc337092-metrics-certs\") pod \"maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p\" (UID: \"982a04b0-d1b7-4f74-ab18-f84ebc337092\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"
Nov 03 22:17:40.105790 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:40.105750    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-f3e839-serving-cert\""
Nov 03 22:17:40.186974 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:40.186936    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fmkv\" (UniqueName: \"kubernetes.io/projected/982a04b0-d1b7-4f74-ab18-f84ebc337092-kube-api-access-2fmkv\") pod \"maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p\" (UID: \"982a04b0-d1b7-4f74-ab18-f84ebc337092\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"
Nov 03 22:17:40.187191 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:40.186988    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/982a04b0-d1b7-4f74-ab18-f84ebc337092-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p\" (UID: \"982a04b0-d1b7-4f74-ab18-f84ebc337092\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"
Nov 03 22:17:40.187191 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:40.187007    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982a04b0-d1b7-4f74-ab18-f84ebc337092-metrics-certs\") pod \"maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p\" (UID: \"982a04b0-d1b7-4f74-ab18-f84ebc337092\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"
Nov 03 22:17:40.187191 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:17:40.187116    5053 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-serving-cert: secret "maas-api-e2e-extmod-a-f3e839-serving-cert" not found
Nov 03 22:17:40.187448 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:17:40.187213    5053 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/982a04b0-d1b7-4f74-ab18-f84ebc337092-maas-api-tls podName:982a04b0-d1b7-4f74-ab18-f84ebc337092 nodeName:}" failed. No retries permitted until 2026-11-03 22:17:40.687191943 +0000 UTC m=+3775.181113804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/982a04b0-d1b7-4f74-ab18-f84ebc337092-maas-api-tls") pod "maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p" (UID: "982a04b0-d1b7-4f74-ab18-f84ebc337092") : secret "maas-api-e2e-extmod-a-f3e839-serving-cert" not found
Nov 03 22:17:40.187448 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:40.187328    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/982a04b0-d1b7-4f74-ab18-f84ebc337092-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p\" (UID: \"982a04b0-d1b7-4f74-ab18-f84ebc337092\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"
Nov 03 22:17:40.187919 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:40.187898    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/982a04b0-d1b7-4f74-ab18-f84ebc337092-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p\" (UID: \"982a04b0-d1b7-4f74-ab18-f84ebc337092\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"
Nov 03 22:17:40.189772 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:40.189747    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982a04b0-d1b7-4f74-ab18-f84ebc337092-metrics-certs\") pod \"maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p\" (UID: \"982a04b0-d1b7-4f74-ab18-f84ebc337092\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"
Nov 03 22:17:40.247947 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:40.247913    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fmkv\" (UniqueName: \"kubernetes.io/projected/982a04b0-d1b7-4f74-ab18-f84ebc337092-kube-api-access-2fmkv\") pod \"maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p\" (UID: \"982a04b0-d1b7-4f74-ab18-f84ebc337092\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"
Nov 03 22:17:40.696170 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:40.696123    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/982a04b0-d1b7-4f74-ab18-f84ebc337092-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p\" (UID: \"982a04b0-d1b7-4f74-ab18-f84ebc337092\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"
Nov 03 22:17:40.698899 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:40.698878    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/982a04b0-d1b7-4f74-ab18-f84ebc337092-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p\" (UID: \"982a04b0-d1b7-4f74-ab18-f84ebc337092\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"
Nov 03 22:17:40.918781 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:40.918748    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"
Nov 03 22:17:40.932499 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:17:40.932453    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:17:41.298487 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:41.298461    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"]
Nov 03 22:17:41.299333 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:17:41.299309    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982a04b0_d1b7_4f74_ab18_f84ebc337092.slice/crio-b3c2269dbbdff3e5c84ecae96fa7aa9382a9a89ba66af05116ba53813a3ab468 WatchSource:0}: Error finding container b3c2269dbbdff3e5c84ecae96fa7aa9382a9a89ba66af05116ba53813a3ab468: Status 404 returned error can't find the container with id b3c2269dbbdff3e5c84ecae96fa7aa9382a9a89ba66af05116ba53813a3ab468
Nov 03 22:17:41.797347 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:41.797299    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p" event={"ID":"982a04b0-d1b7-4f74-ab18-f84ebc337092","Type":"ContainerStarted","Data":"b3c2269dbbdff3e5c84ecae96fa7aa9382a9a89ba66af05116ba53813a3ab468"}
Nov 03 22:17:42.805907 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:42.805866    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p" event={"ID":"982a04b0-d1b7-4f74-ab18-f84ebc337092","Type":"ContainerStarted","Data":"2e7525c3701c6eb6653427ba8e0e3ca82b97d9d62e0b83608a642eb0a00d4d87"}
Nov 03 22:17:42.806461 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:42.806063    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"
Nov 03 22:17:42.919218 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:42.919102    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p" podStartSLOduration=3.079093824 podStartE2EDuration="3.919082873s" totalImagesPullingTime="839.989049ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:42.918438616 +0000 UTC m=+3777.412360491" watchObservedRunningTime="2026-11-03 22:17:42.919082873 +0000 UTC m=+3777.413004749"
Nov 03 22:17:48.819813 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:48.819779    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"
Nov 03 22:17:57.700297 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:17:57.700262    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"]
Nov 03 22:18:08.012758 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:08.012723    5053 generic.go:356] "Generic (PLEG): container finished" podID="3d8d06b2-a794-4262-855d-75c8e268f3b5" containerID="03b5ef7c336998e491f6a28e6c89392f47ee3acf52e32c09168151623d0485bd" exitCode=0
Nov 03 22:18:08.013369 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:08.012801    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29" event={"ID":"3d8d06b2-a794-4262-855d-75c8e268f3b5","Type":"ContainerDied","Data":"03b5ef7c336998e491f6a28e6c89392f47ee3acf52e32c09168151623d0485bd"}
Nov 03 22:18:08.013369 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:08.013308    5053 scope.go:122] "RemoveContainer" containerID="03b5ef7c336998e491f6a28e6c89392f47ee3acf52e32c09168151623d0485bd"
Nov 03 22:18:08.015116 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:08.015085    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:18:08.015198 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:08.015158    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:18:08.015323 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:08.015197    5053 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 22:18:09.043479 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:09.043435    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29" event={"ID":"3d8d06b2-a794-4262-855d-75c8e268f3b5","Type":"ContainerStarted","Data":"1e600356200217fed5681687e02d315fd0702a4a736f9ad8189784ba98310fe1"}
Nov 03 22:18:09.063465 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:09.063432    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:18:09.064877 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:09.064852    5053 patch_prober.go:29] interesting pod/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Nov 03 22:18:09.065007 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:09.064917    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29" podUID="3d8d06b2-a794-4262-855d-75c8e268f3b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Nov 03 22:18:10.063596 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:10.063562    5053 patch_prober.go:29] interesting pod/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Nov 03 22:18:10.064142 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:10.063628    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29" podUID="3d8d06b2-a794-4262-855d-75c8e268f3b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Nov 03 22:18:10.990267 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:18:10.990216    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:18:11.063333 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:11.063302    5053 patch_prober.go:29] interesting pod/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Nov 03 22:18:11.063540 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:11.063375    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29" podUID="3d8d06b2-a794-4262-855d-75c8e268f3b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Nov 03 22:18:11.125980 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:11.125945    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:18:12.064870 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:12.064725    5053 patch_prober.go:29] interesting pod/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Nov 03 22:18:12.064870 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:12.064789    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29" podUID="3d8d06b2-a794-4262-855d-75c8e268f3b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Nov 03 22:18:12.729514 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:12.729473    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48" podUID="e9b7982f-9598-4fb0-b1d6-10a59db8b02b" containerName="main" containerID="cri-o://f6dced47f640e3c6536b9ceb298285ec2419515f209a3b4790e05147348583de" gracePeriod=45
Nov 03 22:18:13.063534 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.063499    5053 patch_prober.go:29] interesting pod/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Nov 03 22:18:13.063595 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.063565    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29" podUID="3d8d06b2-a794-4262-855d-75c8e268f3b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Nov 03 22:18:13.075126 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.075098    5053 generic.go:356] "Generic (PLEG): container finished" podID="e9b7982f-9598-4fb0-b1d6-10a59db8b02b" containerID="f6dced47f640e3c6536b9ceb298285ec2419515f209a3b4790e05147348583de" exitCode=0
Nov 03 22:18:13.075324 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.075181    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48" event={"ID":"e9b7982f-9598-4fb0-b1d6-10a59db8b02b","Type":"ContainerDied","Data":"f6dced47f640e3c6536b9ceb298285ec2419515f209a3b4790e05147348583de"}
Nov 03 22:18:13.075324 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.075254    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48" event={"ID":"e9b7982f-9598-4fb0-b1d6-10a59db8b02b","Type":"ContainerDied","Data":"4b489f671701c6fe3c3213299caedc3a51f9fd9def1aae561d7db0557b82b6df"}
Nov 03 22:18:13.075324 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.075277    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b489f671701c6fe3c3213299caedc3a51f9fd9def1aae561d7db0557b82b6df"
Nov 03 22:18:13.084618 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.084599    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:18:13.171294 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.171229    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-kube-api-access-tplnx\" (UniqueName: \"kubernetes.io/projected/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-kube-api-access-tplnx\") pod \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\" (UID: \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\") "
Nov 03 22:18:13.173102 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.173078    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-dshm\") pod \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\" (UID: \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\") "
Nov 03 22:18:13.173304 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.173287    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-tls-certs\") pod \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\" (UID: \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\") "
Nov 03 22:18:13.173896 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.173874    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-tmp-dir\") pod \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\" (UID: \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\") "
Nov 03 22:18:13.174506 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.174483    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-model-cache\") pod \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\" (UID: \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\") "
Nov 03 22:18:13.174611 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.174567    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-home\" (UniqueName: \"kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-home\") pod \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\" (UID: \"e9b7982f-9598-4fb0-b1d6-10a59db8b02b\") "
Nov 03 22:18:13.175512 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.174296    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-tmp-dir" pod "e9b7982f-9598-4fb0-b1d6-10a59db8b02b" (UID: "e9b7982f-9598-4fb0-b1d6-10a59db8b02b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:18:13.175605 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.175474    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-model-cache" pod "e9b7982f-9598-4fb0-b1d6-10a59db8b02b" (UID: "e9b7982f-9598-4fb0-b1d6-10a59db8b02b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:18:13.175605 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.175571    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-home" pod "e9b7982f-9598-4fb0-b1d6-10a59db8b02b" (UID: "e9b7982f-9598-4fb0-b1d6-10a59db8b02b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:18:13.176172 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.176151    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-kube-api-access-tplnx" pod "e9b7982f-9598-4fb0-b1d6-10a59db8b02b" (UID: "e9b7982f-9598-4fb0-b1d6-10a59db8b02b"). InnerVolumeSpecName "kube-api-access-tplnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:18:13.176521 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.176498    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-tls-certs" pod "e9b7982f-9598-4fb0-b1d6-10a59db8b02b" (UID: "e9b7982f-9598-4fb0-b1d6-10a59db8b02b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:18:13.178941 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.178877    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-dshm" pod "e9b7982f-9598-4fb0-b1d6-10a59db8b02b" (UID: "e9b7982f-9598-4fb0-b1d6-10a59db8b02b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:18:13.276469 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.276432    5053 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-tls-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:18:13.276469 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.276463    5053 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-tmp-dir\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:18:13.276469 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.276472    5053 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-model-cache\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:18:13.276469 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.276480    5053 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-home\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:18:13.276759 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.276490    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tplnx\" (UniqueName: \"kubernetes.io/projected/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-kube-api-access-tplnx\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:18:13.276759 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:13.276499    5053 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9b7982f-9598-4fb0-b1d6-10a59db8b02b-dshm\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:18:14.064100 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:14.064065    5053 patch_prober.go:29] interesting pod/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Nov 03 22:18:14.064565 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:14.064124    5053 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29" podUID="3d8d06b2-a794-4262-855d-75c8e268f3b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Nov 03 22:18:14.069272 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:14.069222    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48" podUID="e9b7982f-9598-4fb0-b1d6-10a59db8b02b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.240:8000/ready\": context deadline exceeded"
Nov 03 22:18:14.081607 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:14.081570    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"
Nov 03 22:18:14.245365 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:14.245332    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"]
Nov 03 22:18:14.285773 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:14.285742    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fb477/auth-test-model-7fb477-kserve-57d9d7b457-jhl48"]
Nov 03 22:18:15.068944 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:15.068915    5053 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:18:15.070124 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:15.070098    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:18:15.669776 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:15.669730    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9b7982f-9598-4fb0-b1d6-10a59db8b02b" path="/var/lib/kubelet/pods/e9b7982f-9598-4fb0-b1d6-10a59db8b02b/volumes"
Nov 03 22:18:40.930786 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:18:40.930747    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:18:48.278412 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:18:48.278379    5053 scope.go:122] "RemoveContainer" containerID="f6dced47f640e3c6536b9ceb298285ec2419515f209a3b4790e05147348583de"
Nov 03 22:19:10.933719 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:19:10.933675    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:19:27.626606 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:27.626566    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx"]
Nov 03 22:19:27.627187 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:27.626763    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx" podUID="a6d529fb-027e-4ce2-af6c-50a7f716ddb9" containerName="maas-api" containerID="cri-o://5c16d72246dda09fb29d9702dea7e75e783339abd1d2dd915745e0a75664c847" gracePeriod=30
Nov 03 22:19:27.929095 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:27.929067    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx"
Nov 03 22:19:28.083826 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:28.083793    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-openshift-service-ca\") pod \"a6d529fb-027e-4ce2-af6c-50a7f716ddb9\" (UID: \"a6d529fb-027e-4ce2-af6c-50a7f716ddb9\") "
Nov 03 22:19:28.084043 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:28.083879    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-kube-api-access-9bw8z\" (UniqueName: \"kubernetes.io/projected/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-kube-api-access-9bw8z\") pod \"a6d529fb-027e-4ce2-af6c-50a7f716ddb9\" (UID: \"a6d529fb-027e-4ce2-af6c-50a7f716ddb9\") "
Nov 03 22:19:28.084043 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:28.083965    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-metrics-certs\") pod \"a6d529fb-027e-4ce2-af6c-50a7f716ddb9\" (UID: \"a6d529fb-027e-4ce2-af6c-50a7f716ddb9\") "
Nov 03 22:19:28.084043 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:28.084036    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-maas-api-tls\") pod \"a6d529fb-027e-4ce2-af6c-50a7f716ddb9\" (UID: \"a6d529fb-027e-4ce2-af6c-50a7f716ddb9\") "
Nov 03 22:19:28.084268 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:28.084190    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-openshift-service-ca" pod "a6d529fb-027e-4ce2-af6c-50a7f716ddb9" (UID: "a6d529fb-027e-4ce2-af6c-50a7f716ddb9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:19:28.084762 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:28.084683    5053 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-openshift-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:28.087071 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:28.087041    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-metrics-certs" pod "a6d529fb-027e-4ce2-af6c-50a7f716ddb9" (UID: "a6d529fb-027e-4ce2-af6c-50a7f716ddb9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:19:28.087308 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:28.087282    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-kube-api-access-9bw8z" pod "a6d529fb-027e-4ce2-af6c-50a7f716ddb9" (UID: "a6d529fb-027e-4ce2-af6c-50a7f716ddb9"). InnerVolumeSpecName "kube-api-access-9bw8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:19:28.087428 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:28.087410    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-maas-api-tls" pod "a6d529fb-027e-4ce2-af6c-50a7f716ddb9" (UID: "a6d529fb-027e-4ce2-af6c-50a7f716ddb9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:19:28.186546 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:28.186472    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bw8z\" (UniqueName: \"kubernetes.io/projected/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-kube-api-access-9bw8z\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:28.186546 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:28.186504    5053 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-metrics-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:28.186546 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:28.186514    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6d529fb-027e-4ce2-af6c-50a7f716ddb9-maas-api-tls\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:28.645129 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:28.645079    5053 generic.go:356] "Generic (PLEG): container finished" podID="a6d529fb-027e-4ce2-af6c-50a7f716ddb9" containerID="5c16d72246dda09fb29d9702dea7e75e783339abd1d2dd915745e0a75664c847" exitCode=0
Nov 03 22:19:28.645705 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:28.645145    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx"
Nov 03 22:19:28.645705 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:28.645208    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx" event={"ID":"a6d529fb-027e-4ce2-af6c-50a7f716ddb9","Type":"ContainerDied","Data":"5c16d72246dda09fb29d9702dea7e75e783339abd1d2dd915745e0a75664c847"}
Nov 03 22:19:28.645705 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:28.645310    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx" event={"ID":"a6d529fb-027e-4ce2-af6c-50a7f716ddb9","Type":"ContainerDied","Data":"89fbb74cffe802722e8e20b301026dccf862fde3c2a319aa4e0912db7fab2836"}
Nov 03 22:19:28.645705 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:28.645363    5053 scope.go:122] "RemoveContainer" containerID="5c16d72246dda09fb29d9702dea7e75e783339abd1d2dd915745e0a75664c847"
Nov 03 22:19:28.660553 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:28.660530    5053 scope.go:122] "RemoveContainer" containerID="5c16d72246dda09fb29d9702dea7e75e783339abd1d2dd915745e0a75664c847"
Nov 03 22:19:28.660890 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:19:28.660869    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c16d72246dda09fb29d9702dea7e75e783339abd1d2dd915745e0a75664c847\": container with ID starting with 5c16d72246dda09fb29d9702dea7e75e783339abd1d2dd915745e0a75664c847 not found: ID does not exist" containerID="5c16d72246dda09fb29d9702dea7e75e783339abd1d2dd915745e0a75664c847"
Nov 03 22:19:28.660943 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:28.660901    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c16d72246dda09fb29d9702dea7e75e783339abd1d2dd915745e0a75664c847"} err="failed to get container status \"5c16d72246dda09fb29d9702dea7e75e783339abd1d2dd915745e0a75664c847\": rpc error: code = NotFound desc = could not find container \"5c16d72246dda09fb29d9702dea7e75e783339abd1d2dd915745e0a75664c847\": container with ID starting with 5c16d72246dda09fb29d9702dea7e75e783339abd1d2dd915745e0a75664c847 not found: ID does not exist"
Nov 03 22:19:28.808495 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:28.808461    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx"]
Nov 03 22:19:28.858485 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:28.858452    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx"]
Nov 03 22:19:29.667617 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:29.667585    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6d529fb-027e-4ce2-af6c-50a7f716ddb9" path="/var/lib/kubelet/pods/a6d529fb-027e-4ce2-af6c-50a7f716ddb9/volumes"
Nov 03 22:19:30.863024 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:30.862974    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5b1489-548fccb58f-c7fmx" podUID="a6d529fb-027e-4ce2-af6c-50a7f716ddb9" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.242:8443/health\": context deadline exceeded"
Nov 03 22:19:36.173766 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:36.173728    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"]
Nov 03 22:19:36.174263 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:36.174006    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk" podUID="4f17d174-9d7f-4b54-82c5-6c4dac27b48b" containerName="istio-proxy" containerID="cri-o://58af5eaa3e828c811720dc448ef8afe679b1da811eb26441d18cb8f10d16cf1f" gracePeriod=30
Nov 03 22:19:37.285804 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:37.285768    5053 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:19:37.286219 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:37.285845    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk" podUID="4f17d174-9d7f-4b54-82c5-6c4dac27b48b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:19:40.930323 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:19:40.930285    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:19:41.466618 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.466590    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:19:41.566116 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.566079    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-kube-api-access-qvnpr\" (UniqueName: \"kubernetes.io/projected/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-kube-api-access-qvnpr\") pod \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") "
Nov 03 22:19:41.566116 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.566120    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istiod-ca-cert\") pod \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") "
Nov 03 22:19:41.566450 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.566155    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-workload-certs\") pod \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") "
Nov 03 22:19:41.566450 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.566204    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-envoy\") pod \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") "
Nov 03 22:19:41.566450 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.566281    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-podinfo\") pod \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") "
Nov 03 22:19:41.566450 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.566332    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-token\" (UniqueName: \"kubernetes.io/projected/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-token\") pod \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") "
Nov 03 22:19:41.566450 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.566368    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-credential-socket\") pod \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") "
Nov 03 22:19:41.566733 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.566458    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-workload-socket\") pod \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") "
Nov 03 22:19:41.566733 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.566536    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-workload-certs" pod "4f17d174-9d7f-4b54-82c5-6c4dac27b48b" (UID: "4f17d174-9d7f-4b54-82c5-6c4dac27b48b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:19:41.566733 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.566552    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-data\") pod \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\" (UID: \"4f17d174-9d7f-4b54-82c5-6c4dac27b48b\") "
Nov 03 22:19:41.566733 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.566625    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istiod-ca-cert" pod "4f17d174-9d7f-4b54-82c5-6c4dac27b48b" (UID: "4f17d174-9d7f-4b54-82c5-6c4dac27b48b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:19:41.566951 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.566768    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-credential-socket" pod "4f17d174-9d7f-4b54-82c5-6c4dac27b48b" (UID: "4f17d174-9d7f-4b54-82c5-6c4dac27b48b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:19:41.567010 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.566929    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-workload-socket" pod "4f17d174-9d7f-4b54-82c5-6c4dac27b48b" (UID: "4f17d174-9d7f-4b54-82c5-6c4dac27b48b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:19:41.567010 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.566963    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-data" pod "4f17d174-9d7f-4b54-82c5-6c4dac27b48b" (UID: "4f17d174-9d7f-4b54-82c5-6c4dac27b48b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:19:41.567102 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.567025    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:41.567102 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.567063    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:41.567102 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.567081    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:41.568793 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.568762    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-podinfo" pod "4f17d174-9d7f-4b54-82c5-6c4dac27b48b" (UID: "4f17d174-9d7f-4b54-82c5-6c4dac27b48b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:19:41.568990 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.568972    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-token" pod "4f17d174-9d7f-4b54-82c5-6c4dac27b48b" (UID: "4f17d174-9d7f-4b54-82c5-6c4dac27b48b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:19:41.569076 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.569057    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-envoy" pod "4f17d174-9d7f-4b54-82c5-6c4dac27b48b" (UID: "4f17d174-9d7f-4b54-82c5-6c4dac27b48b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:19:41.569133 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.569114    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-kube-api-access-qvnpr" pod "4f17d174-9d7f-4b54-82c5-6c4dac27b48b" (UID: "4f17d174-9d7f-4b54-82c5-6c4dac27b48b"). InnerVolumeSpecName "kube-api-access-qvnpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:19:41.668293 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.668257    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:41.668293 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.668292    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:41.668598 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.668306    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:41.668598 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.668323    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvnpr\" (UniqueName: \"kubernetes.io/projected/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-kube-api-access-qvnpr\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:41.668598 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.668338    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:41.668598 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.668351    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f17d174-9d7f-4b54-82c5-6c4dac27b48b-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:41.739777 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.739740    5053 generic.go:356] "Generic (PLEG): container finished" podID="4f17d174-9d7f-4b54-82c5-6c4dac27b48b" containerID="58af5eaa3e828c811720dc448ef8afe679b1da811eb26441d18cb8f10d16cf1f" exitCode=0
Nov 03 22:19:41.739968 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.739793    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"
Nov 03 22:19:41.739968 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.739826    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk" event={"ID":"4f17d174-9d7f-4b54-82c5-6c4dac27b48b","Type":"ContainerDied","Data":"58af5eaa3e828c811720dc448ef8afe679b1da811eb26441d18cb8f10d16cf1f"}
Nov 03 22:19:41.739968 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.739884    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk" event={"ID":"4f17d174-9d7f-4b54-82c5-6c4dac27b48b","Type":"ContainerDied","Data":"48d799c7785a7f70ab4469d4363b6b828824bc65e5f02bd3a7a994549634dec3"}
Nov 03 22:19:41.739968 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.739913    5053 scope.go:122] "RemoveContainer" containerID="58af5eaa3e828c811720dc448ef8afe679b1da811eb26441d18cb8f10d16cf1f"
Nov 03 22:19:41.750282 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.750229    5053 scope.go:122] "RemoveContainer" containerID="1ee2cd4dca86f76a4e71731677982d4a651f35d518a258a82243b0e1ce32b22e"
Nov 03 22:19:41.760429 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.760403    5053 scope.go:122] "RemoveContainer" containerID="58af5eaa3e828c811720dc448ef8afe679b1da811eb26441d18cb8f10d16cf1f"
Nov 03 22:19:41.760730 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:19:41.760699    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58af5eaa3e828c811720dc448ef8afe679b1da811eb26441d18cb8f10d16cf1f\": container with ID starting with 58af5eaa3e828c811720dc448ef8afe679b1da811eb26441d18cb8f10d16cf1f not found: ID does not exist" containerID="58af5eaa3e828c811720dc448ef8afe679b1da811eb26441d18cb8f10d16cf1f"
Nov 03 22:19:41.760861 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.760734    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58af5eaa3e828c811720dc448ef8afe679b1da811eb26441d18cb8f10d16cf1f"} err="failed to get container status \"58af5eaa3e828c811720dc448ef8afe679b1da811eb26441d18cb8f10d16cf1f\": rpc error: code = NotFound desc = could not find container \"58af5eaa3e828c811720dc448ef8afe679b1da811eb26441d18cb8f10d16cf1f\": container with ID starting with 58af5eaa3e828c811720dc448ef8afe679b1da811eb26441d18cb8f10d16cf1f not found: ID does not exist"
Nov 03 22:19:41.760861 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.760755    5053 scope.go:122] "RemoveContainer" containerID="1ee2cd4dca86f76a4e71731677982d4a651f35d518a258a82243b0e1ce32b22e"
Nov 03 22:19:41.761056 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:19:41.761022    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ee2cd4dca86f76a4e71731677982d4a651f35d518a258a82243b0e1ce32b22e\": container with ID starting with 1ee2cd4dca86f76a4e71731677982d4a651f35d518a258a82243b0e1ce32b22e not found: ID does not exist" containerID="1ee2cd4dca86f76a4e71731677982d4a651f35d518a258a82243b0e1ce32b22e"
Nov 03 22:19:41.761152 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.761065    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ee2cd4dca86f76a4e71731677982d4a651f35d518a258a82243b0e1ce32b22e"} err="failed to get container status \"1ee2cd4dca86f76a4e71731677982d4a651f35d518a258a82243b0e1ce32b22e\": rpc error: code = NotFound desc = could not find container \"1ee2cd4dca86f76a4e71731677982d4a651f35d518a258a82243b0e1ce32b22e\": container with ID starting with 1ee2cd4dca86f76a4e71731677982d4a651f35d518a258a82243b0e1ce32b22e not found: ID does not exist"
Nov 03 22:19:41.904276 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.904218    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"]
Nov 03 22:19:41.907431 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:41.907402    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5b1489-openshift-default-654bb5f58c-g96fk"]
Nov 03 22:19:43.669133 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:43.669089    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f17d174-9d7f-4b54-82c5-6c4dac27b48b" path="/var/lib/kubelet/pods/4f17d174-9d7f-4b54-82c5-6c4dac27b48b/volumes"
Nov 03 22:19:46.334805 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:46.334776    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 22:19:46.341438 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:46.341407    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 22:19:46.344664 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:46.344636    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 22:19:46.350392 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:19:46.350370    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 22:20:10.930609 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:20:10.930570    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:20:13.372374 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:13.372336    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn"]
Nov 03 22:20:13.372864 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:13.372549    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn" podUID="722a009c-bbee-44ea-b624-bd8eeb0897b8" containerName="maas-api" containerID="cri-o://72c177d59490bdba1e17001749183d94e3afe4a7c2d7a364492224d0801162ea" gracePeriod=30
Nov 03 22:20:13.679067 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:13.679036    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn"
Nov 03 22:20:13.793664 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:13.793566    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/722a009c-bbee-44ea-b624-bd8eeb0897b8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/722a009c-bbee-44ea-b624-bd8eeb0897b8-openshift-service-ca\") pod \"722a009c-bbee-44ea-b624-bd8eeb0897b8\" (UID: \"722a009c-bbee-44ea-b624-bd8eeb0897b8\") "
Nov 03 22:20:13.794801 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:13.794778    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/722a009c-bbee-44ea-b624-bd8eeb0897b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/722a009c-bbee-44ea-b624-bd8eeb0897b8-metrics-certs\") pod \"722a009c-bbee-44ea-b624-bd8eeb0897b8\" (UID: \"722a009c-bbee-44ea-b624-bd8eeb0897b8\") "
Nov 03 22:20:13.795015 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:13.794702    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/722a009c-bbee-44ea-b624-bd8eeb0897b8-openshift-service-ca" pod "722a009c-bbee-44ea-b624-bd8eeb0897b8" (UID: "722a009c-bbee-44ea-b624-bd8eeb0897b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:20:13.795015 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:13.794887    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/722a009c-bbee-44ea-b624-bd8eeb0897b8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/722a009c-bbee-44ea-b624-bd8eeb0897b8-maas-api-tls\") pod \"722a009c-bbee-44ea-b624-bd8eeb0897b8\" (UID: \"722a009c-bbee-44ea-b624-bd8eeb0897b8\") "
Nov 03 22:20:13.795015 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:13.794948    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/722a009c-bbee-44ea-b624-bd8eeb0897b8-kube-api-access-wbplp\" (UniqueName: \"kubernetes.io/projected/722a009c-bbee-44ea-b624-bd8eeb0897b8-kube-api-access-wbplp\") pod \"722a009c-bbee-44ea-b624-bd8eeb0897b8\" (UID: \"722a009c-bbee-44ea-b624-bd8eeb0897b8\") "
Nov 03 22:20:13.796707 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:13.795628    5053 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/722a009c-bbee-44ea-b624-bd8eeb0897b8-openshift-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:13.798470 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:13.798440    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/722a009c-bbee-44ea-b624-bd8eeb0897b8-metrics-certs" pod "722a009c-bbee-44ea-b624-bd8eeb0897b8" (UID: "722a009c-bbee-44ea-b624-bd8eeb0897b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:20:13.799373 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:13.798819    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/722a009c-bbee-44ea-b624-bd8eeb0897b8-maas-api-tls" pod "722a009c-bbee-44ea-b624-bd8eeb0897b8" (UID: "722a009c-bbee-44ea-b624-bd8eeb0897b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:20:13.799373 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:13.799130    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722a009c-bbee-44ea-b624-bd8eeb0897b8-kube-api-access-wbplp" pod "722a009c-bbee-44ea-b624-bd8eeb0897b8" (UID: "722a009c-bbee-44ea-b624-bd8eeb0897b8"). InnerVolumeSpecName "kube-api-access-wbplp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:20:13.898476 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:13.898434    5053 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/722a009c-bbee-44ea-b624-bd8eeb0897b8-metrics-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:13.898476 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:13.898469    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/722a009c-bbee-44ea-b624-bd8eeb0897b8-maas-api-tls\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:13.898476 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:13.898483    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbplp\" (UniqueName: \"kubernetes.io/projected/722a009c-bbee-44ea-b624-bd8eeb0897b8-kube-api-access-wbplp\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:13.967137 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:13.967100    5053 generic.go:356] "Generic (PLEG): container finished" podID="722a009c-bbee-44ea-b624-bd8eeb0897b8" containerID="72c177d59490bdba1e17001749183d94e3afe4a7c2d7a364492224d0801162ea" exitCode=0
Nov 03 22:20:13.967346 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:13.967145    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn"
Nov 03 22:20:13.967346 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:13.967189    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn" event={"ID":"722a009c-bbee-44ea-b624-bd8eeb0897b8","Type":"ContainerDied","Data":"72c177d59490bdba1e17001749183d94e3afe4a7c2d7a364492224d0801162ea"}
Nov 03 22:20:13.967346 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:13.967281    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn" event={"ID":"722a009c-bbee-44ea-b624-bd8eeb0897b8","Type":"ContainerDied","Data":"98ca7d4621c6f2ee73aeade7fcb1700708a54641996793d5dd2b188c33bce9e1"}
Nov 03 22:20:13.967346 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:13.967309    5053 scope.go:122] "RemoveContainer" containerID="72c177d59490bdba1e17001749183d94e3afe4a7c2d7a364492224d0801162ea"
Nov 03 22:20:13.978288 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:13.978264    5053 scope.go:122] "RemoveContainer" containerID="72c177d59490bdba1e17001749183d94e3afe4a7c2d7a364492224d0801162ea"
Nov 03 22:20:13.978583 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:20:13.978556    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72c177d59490bdba1e17001749183d94e3afe4a7c2d7a364492224d0801162ea\": container with ID starting with 72c177d59490bdba1e17001749183d94e3afe4a7c2d7a364492224d0801162ea not found: ID does not exist" containerID="72c177d59490bdba1e17001749183d94e3afe4a7c2d7a364492224d0801162ea"
Nov 03 22:20:13.978678 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:13.978592    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72c177d59490bdba1e17001749183d94e3afe4a7c2d7a364492224d0801162ea"} err="failed to get container status \"72c177d59490bdba1e17001749183d94e3afe4a7c2d7a364492224d0801162ea\": rpc error: code = NotFound desc = could not find container \"72c177d59490bdba1e17001749183d94e3afe4a7c2d7a364492224d0801162ea\": container with ID starting with 72c177d59490bdba1e17001749183d94e3afe4a7c2d7a364492224d0801162ea not found: ID does not exist"
Nov 03 22:20:14.130831 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:14.130793    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn"]
Nov 03 22:20:14.134097 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:14.134067    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d7b12e-dffbd67db-svxzn"]
Nov 03 22:20:15.669961 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:15.669920    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="722a009c-bbee-44ea-b624-bd8eeb0897b8" path="/var/lib/kubelet/pods/722a009c-bbee-44ea-b624-bd8eeb0897b8/volumes"
Nov 03 22:20:40.930690 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:20:40.930651    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:20:43.188958 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:43.188922    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"]
Nov 03 22:20:43.189582 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:43.189131    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p" podUID="982a04b0-d1b7-4f74-ab18-f84ebc337092" containerName="maas-api" containerID="cri-o://2e7525c3701c6eb6653427ba8e0e3ca82b97d9d62e0b83608a642eb0a00d4d87" gracePeriod=30
Nov 03 22:20:43.492561 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:43.492534    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"
Nov 03 22:20:43.612708 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:43.612674    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/982a04b0-d1b7-4f74-ab18-f84ebc337092-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/982a04b0-d1b7-4f74-ab18-f84ebc337092-maas-api-tls\") pod \"982a04b0-d1b7-4f74-ab18-f84ebc337092\" (UID: \"982a04b0-d1b7-4f74-ab18-f84ebc337092\") "
Nov 03 22:20:43.612898 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:43.612836    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/982a04b0-d1b7-4f74-ab18-f84ebc337092-kube-api-access-2fmkv\" (UniqueName: \"kubernetes.io/projected/982a04b0-d1b7-4f74-ab18-f84ebc337092-kube-api-access-2fmkv\") pod \"982a04b0-d1b7-4f74-ab18-f84ebc337092\" (UID: \"982a04b0-d1b7-4f74-ab18-f84ebc337092\") "
Nov 03 22:20:43.612966 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:43.612936    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/982a04b0-d1b7-4f74-ab18-f84ebc337092-metrics-certs\" (UniqueName: \"kubernetes.io/secret/982a04b0-d1b7-4f74-ab18-f84ebc337092-metrics-certs\") pod \"982a04b0-d1b7-4f74-ab18-f84ebc337092\" (UID: \"982a04b0-d1b7-4f74-ab18-f84ebc337092\") "
Nov 03 22:20:43.613040 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:43.612987    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/982a04b0-d1b7-4f74-ab18-f84ebc337092-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/982a04b0-d1b7-4f74-ab18-f84ebc337092-openshift-service-ca\") pod \"982a04b0-d1b7-4f74-ab18-f84ebc337092\" (UID: \"982a04b0-d1b7-4f74-ab18-f84ebc337092\") "
Nov 03 22:20:43.613631 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:43.613595    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/982a04b0-d1b7-4f74-ab18-f84ebc337092-openshift-service-ca" pod "982a04b0-d1b7-4f74-ab18-f84ebc337092" (UID: "982a04b0-d1b7-4f74-ab18-f84ebc337092"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:20:43.616338 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:43.616293    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/982a04b0-d1b7-4f74-ab18-f84ebc337092-metrics-certs" pod "982a04b0-d1b7-4f74-ab18-f84ebc337092" (UID: "982a04b0-d1b7-4f74-ab18-f84ebc337092"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:20:43.616670 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:43.616643    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/982a04b0-d1b7-4f74-ab18-f84ebc337092-maas-api-tls" pod "982a04b0-d1b7-4f74-ab18-f84ebc337092" (UID: "982a04b0-d1b7-4f74-ab18-f84ebc337092"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:20:43.617693 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:43.617670    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982a04b0-d1b7-4f74-ab18-f84ebc337092-kube-api-access-2fmkv" pod "982a04b0-d1b7-4f74-ab18-f84ebc337092" (UID: "982a04b0-d1b7-4f74-ab18-f84ebc337092"). InnerVolumeSpecName "kube-api-access-2fmkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:20:43.714932 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:43.714836    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/982a04b0-d1b7-4f74-ab18-f84ebc337092-maas-api-tls\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:43.714932 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:43.714871    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fmkv\" (UniqueName: \"kubernetes.io/projected/982a04b0-d1b7-4f74-ab18-f84ebc337092-kube-api-access-2fmkv\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:43.714932 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:43.714886    5053 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982a04b0-d1b7-4f74-ab18-f84ebc337092-metrics-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:43.714932 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:43.714902    5053 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/982a04b0-d1b7-4f74-ab18-f84ebc337092-openshift-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:44.183492 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:44.183384    5053 generic.go:356] "Generic (PLEG): container finished" podID="982a04b0-d1b7-4f74-ab18-f84ebc337092" containerID="2e7525c3701c6eb6653427ba8e0e3ca82b97d9d62e0b83608a642eb0a00d4d87" exitCode=0
Nov 03 22:20:44.183492 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:44.183452    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"
Nov 03 22:20:44.183806 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:44.183507    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p" event={"ID":"982a04b0-d1b7-4f74-ab18-f84ebc337092","Type":"ContainerDied","Data":"2e7525c3701c6eb6653427ba8e0e3ca82b97d9d62e0b83608a642eb0a00d4d87"}
Nov 03 22:20:44.183806 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:44.183547    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p" event={"ID":"982a04b0-d1b7-4f74-ab18-f84ebc337092","Type":"ContainerDied","Data":"b3c2269dbbdff3e5c84ecae96fa7aa9382a9a89ba66af05116ba53813a3ab468"}
Nov 03 22:20:44.183806 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:44.183566    5053 scope.go:122] "RemoveContainer" containerID="2e7525c3701c6eb6653427ba8e0e3ca82b97d9d62e0b83608a642eb0a00d4d87"
Nov 03 22:20:44.196112 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:44.195959    5053 scope.go:122] "RemoveContainer" containerID="2e7525c3701c6eb6653427ba8e0e3ca82b97d9d62e0b83608a642eb0a00d4d87"
Nov 03 22:20:44.196452 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:20:44.196369    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7525c3701c6eb6653427ba8e0e3ca82b97d9d62e0b83608a642eb0a00d4d87\": container with ID starting with 2e7525c3701c6eb6653427ba8e0e3ca82b97d9d62e0b83608a642eb0a00d4d87 not found: ID does not exist" containerID="2e7525c3701c6eb6653427ba8e0e3ca82b97d9d62e0b83608a642eb0a00d4d87"
Nov 03 22:20:44.196452 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:44.196395    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7525c3701c6eb6653427ba8e0e3ca82b97d9d62e0b83608a642eb0a00d4d87"} err="failed to get container status \"2e7525c3701c6eb6653427ba8e0e3ca82b97d9d62e0b83608a642eb0a00d4d87\": rpc error: code = NotFound desc = could not find container \"2e7525c3701c6eb6653427ba8e0e3ca82b97d9d62e0b83608a642eb0a00d4d87\": container with ID starting with 2e7525c3701c6eb6653427ba8e0e3ca82b97d9d62e0b83608a642eb0a00d4d87 not found: ID does not exist"
Nov 03 22:20:44.349767 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:44.349736    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"]
Nov 03 22:20:44.352519 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:44.352495    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f3e839-5dcf5bf7c-t5s5p"]
Nov 03 22:20:45.668670 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:45.668632    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="982a04b0-d1b7-4f74-ab18-f84ebc337092" path="/var/lib/kubelet/pods/982a04b0-d1b7-4f74-ab18-f84ebc337092/volumes"
Nov 03 22:20:51.972340 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:51.972305    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"]
Nov 03 22:20:51.972799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:51.972565    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29" podUID="3d8d06b2-a794-4262-855d-75c8e268f3b5" containerName="istio-proxy" containerID="cri-o://1e600356200217fed5681687e02d315fd0702a4a736f9ad8189784ba98310fe1" gracePeriod=30
Nov 03 22:20:57.259143 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.259116    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:20:57.273366 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.273329    5053 generic.go:356] "Generic (PLEG): container finished" podID="3d8d06b2-a794-4262-855d-75c8e268f3b5" containerID="1e600356200217fed5681687e02d315fd0702a4a736f9ad8189784ba98310fe1" exitCode=0
Nov 03 22:20:57.273568 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.273378    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29" event={"ID":"3d8d06b2-a794-4262-855d-75c8e268f3b5","Type":"ContainerDied","Data":"1e600356200217fed5681687e02d315fd0702a4a736f9ad8189784ba98310fe1"}
Nov 03 22:20:57.273568 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.273419    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29" event={"ID":"3d8d06b2-a794-4262-855d-75c8e268f3b5","Type":"ContainerDied","Data":"f7e6dbceaa8ca0ff29366ff48c74a678a0ac8fd755531187609d0fd284134ed2"}
Nov 03 22:20:57.273568 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.273442    5053 scope.go:122] "RemoveContainer" containerID="1e600356200217fed5681687e02d315fd0702a4a736f9ad8189784ba98310fe1"
Nov 03 22:20:57.273568 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.273381    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"
Nov 03 22:20:57.283985 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.283958    5053 scope.go:122] "RemoveContainer" containerID="03b5ef7c336998e491f6a28e6c89392f47ee3acf52e32c09168151623d0485bd"
Nov 03 22:20:57.296694 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.296609    5053 scope.go:122] "RemoveContainer" containerID="1e600356200217fed5681687e02d315fd0702a4a736f9ad8189784ba98310fe1"
Nov 03 22:20:57.297080 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:20:57.297051    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e600356200217fed5681687e02d315fd0702a4a736f9ad8189784ba98310fe1\": container with ID starting with 1e600356200217fed5681687e02d315fd0702a4a736f9ad8189784ba98310fe1 not found: ID does not exist" containerID="1e600356200217fed5681687e02d315fd0702a4a736f9ad8189784ba98310fe1"
Nov 03 22:20:57.297177 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.297092    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e600356200217fed5681687e02d315fd0702a4a736f9ad8189784ba98310fe1"} err="failed to get container status \"1e600356200217fed5681687e02d315fd0702a4a736f9ad8189784ba98310fe1\": rpc error: code = NotFound desc = could not find container \"1e600356200217fed5681687e02d315fd0702a4a736f9ad8189784ba98310fe1\": container with ID starting with 1e600356200217fed5681687e02d315fd0702a4a736f9ad8189784ba98310fe1 not found: ID does not exist"
Nov 03 22:20:57.297177 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.297119    5053 scope.go:122] "RemoveContainer" containerID="03b5ef7c336998e491f6a28e6c89392f47ee3acf52e32c09168151623d0485bd"
Nov 03 22:20:57.297585 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:20:57.297561    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03b5ef7c336998e491f6a28e6c89392f47ee3acf52e32c09168151623d0485bd\": container with ID starting with 03b5ef7c336998e491f6a28e6c89392f47ee3acf52e32c09168151623d0485bd not found: ID does not exist" containerID="03b5ef7c336998e491f6a28e6c89392f47ee3acf52e32c09168151623d0485bd"
Nov 03 22:20:57.297632 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.297596    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03b5ef7c336998e491f6a28e6c89392f47ee3acf52e32c09168151623d0485bd"} err="failed to get container status \"03b5ef7c336998e491f6a28e6c89392f47ee3acf52e32c09168151623d0485bd\": rpc error: code = NotFound desc = could not find container \"03b5ef7c336998e491f6a28e6c89392f47ee3acf52e32c09168151623d0485bd\": container with ID starting with 03b5ef7c336998e491f6a28e6c89392f47ee3acf52e32c09168151623d0485bd not found: ID does not exist"
Nov 03 22:20:57.394747 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.394712    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-podinfo\") pod \"3d8d06b2-a794-4262-855d-75c8e268f3b5\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") "
Nov 03 22:20:57.394957 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.394756    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-workload-socket\") pod \"3d8d06b2-a794-4262-855d-75c8e268f3b5\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") "
Nov 03 22:20:57.394957 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.394818    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d8d06b2-a794-4262-855d-75c8e268f3b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d8d06b2-a794-4262-855d-75c8e268f3b5-istiod-ca-cert\") pod \"3d8d06b2-a794-4262-855d-75c8e268f3b5\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") "
Nov 03 22:20:57.394957 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.394840    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-data\") pod \"3d8d06b2-a794-4262-855d-75c8e268f3b5\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") "
Nov 03 22:20:57.394957 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.394920    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-envoy\") pod \"3d8d06b2-a794-4262-855d-75c8e268f3b5\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") "
Nov 03 22:20:57.395185 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.394960    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-token\" (UniqueName: \"kubernetes.io/projected/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-token\") pod \"3d8d06b2-a794-4262-855d-75c8e268f3b5\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") "
Nov 03 22:20:57.395185 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.395014    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d8d06b2-a794-4262-855d-75c8e268f3b5-kube-api-access-lrx2l\" (UniqueName: \"kubernetes.io/projected/3d8d06b2-a794-4262-855d-75c8e268f3b5-kube-api-access-lrx2l\") pod \"3d8d06b2-a794-4262-855d-75c8e268f3b5\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") "
Nov 03 22:20:57.395185 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.395076    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-credential-socket\") pod \"3d8d06b2-a794-4262-855d-75c8e268f3b5\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") "
Nov 03 22:20:57.395185 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.395126    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-workload-certs\") pod \"3d8d06b2-a794-4262-855d-75c8e268f3b5\" (UID: \"3d8d06b2-a794-4262-855d-75c8e268f3b5\") "
Nov 03 22:20:57.395185 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.395136    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-workload-socket" pod "3d8d06b2-a794-4262-855d-75c8e268f3b5" (UID: "3d8d06b2-a794-4262-855d-75c8e268f3b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:20:57.395510 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.395256    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8d06b2-a794-4262-855d-75c8e268f3b5-istiod-ca-cert" pod "3d8d06b2-a794-4262-855d-75c8e268f3b5" (UID: "3d8d06b2-a794-4262-855d-75c8e268f3b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:20:57.395510 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.395256    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-data" pod "3d8d06b2-a794-4262-855d-75c8e268f3b5" (UID: "3d8d06b2-a794-4262-855d-75c8e268f3b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:20:57.395662 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.395635    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-credential-socket" pod "3d8d06b2-a794-4262-855d-75c8e268f3b5" (UID: "3d8d06b2-a794-4262-855d-75c8e268f3b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:20:57.395662 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.395650    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-workload-certs" pod "3d8d06b2-a794-4262-855d-75c8e268f3b5" (UID: "3d8d06b2-a794-4262-855d-75c8e268f3b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:20:57.395915 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.395809    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:57.395915 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.395828    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d8d06b2-a794-4262-855d-75c8e268f3b5-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:57.395915 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.395843    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:57.395915 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.395857    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:57.395915 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.395870    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:57.397343 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.397308    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-podinfo" pod "3d8d06b2-a794-4262-855d-75c8e268f3b5" (UID: "3d8d06b2-a794-4262-855d-75c8e268f3b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:20:57.397535 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.397514    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-token" pod "3d8d06b2-a794-4262-855d-75c8e268f3b5" (UID: "3d8d06b2-a794-4262-855d-75c8e268f3b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:20:57.397627 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.397609    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8d06b2-a794-4262-855d-75c8e268f3b5-kube-api-access-lrx2l" pod "3d8d06b2-a794-4262-855d-75c8e268f3b5" (UID: "3d8d06b2-a794-4262-855d-75c8e268f3b5"). InnerVolumeSpecName "kube-api-access-lrx2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:20:57.397678 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.397661    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-envoy" pod "3d8d06b2-a794-4262-855d-75c8e268f3b5" (UID: "3d8d06b2-a794-4262-855d-75c8e268f3b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:20:57.496942 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.496845    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:57.496942 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.496881    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:57.496942 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.496892    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d8d06b2-a794-4262-855d-75c8e268f3b5-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:57.496942 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.496900    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrx2l\" (UniqueName: \"kubernetes.io/projected/3d8d06b2-a794-4262-855d-75c8e268f3b5-kube-api-access-lrx2l\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:57.695064 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.695020    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"]
Nov 03 22:20:57.699417 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:57.699387    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-f3e839-openshift-default-9ff784989-z2l29"]
Nov 03 22:20:59.667931 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:20:59.667899    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d8d06b2-a794-4262-855d-75c8e268f3b5" path="/var/lib/kubelet/pods/3d8d06b2-a794-4262-855d-75c8e268f3b5/volumes"
Nov 03 22:21:00.519330 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:00.519292    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"]
Nov 03 22:21:00.519600 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:00.519566    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh" podUID="9cbef392-ef4f-475b-a326-d3aaf057d256" containerName="maas-api" containerID="cri-o://c8b0df52bec098d47ac04e7d21d96554c6770b69f68e74485eaae9e090bdeaf0" gracePeriod=30
Nov 03 22:21:00.799323 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:00.799292    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"
Nov 03 22:21:00.956878 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:00.956841    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cbef392-ef4f-475b-a326-d3aaf057d256-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cbef392-ef4f-475b-a326-d3aaf057d256-maas-api-tls\") pod \"9cbef392-ef4f-475b-a326-d3aaf057d256\" (UID: \"9cbef392-ef4f-475b-a326-d3aaf057d256\") "
Nov 03 22:21:00.957089 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:00.956962    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cbef392-ef4f-475b-a326-d3aaf057d256-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cbef392-ef4f-475b-a326-d3aaf057d256-openshift-service-ca\") pod \"9cbef392-ef4f-475b-a326-d3aaf057d256\" (UID: \"9cbef392-ef4f-475b-a326-d3aaf057d256\") "
Nov 03 22:21:00.957089 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:00.957062    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cbef392-ef4f-475b-a326-d3aaf057d256-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cbef392-ef4f-475b-a326-d3aaf057d256-metrics-certs\") pod \"9cbef392-ef4f-475b-a326-d3aaf057d256\" (UID: \"9cbef392-ef4f-475b-a326-d3aaf057d256\") "
Nov 03 22:21:00.957220 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:00.957110    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cbef392-ef4f-475b-a326-d3aaf057d256-kube-api-access-frl6x\" (UniqueName: \"kubernetes.io/projected/9cbef392-ef4f-475b-a326-d3aaf057d256-kube-api-access-frl6x\") pod \"9cbef392-ef4f-475b-a326-d3aaf057d256\" (UID: \"9cbef392-ef4f-475b-a326-d3aaf057d256\") "
Nov 03 22:21:00.957423 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:00.957397    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cbef392-ef4f-475b-a326-d3aaf057d256-openshift-service-ca" pod "9cbef392-ef4f-475b-a326-d3aaf057d256" (UID: "9cbef392-ef4f-475b-a326-d3aaf057d256"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:21:00.957781 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:00.957757    5053 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cbef392-ef4f-475b-a326-d3aaf057d256-openshift-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:00.959516 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:00.959479    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cbef392-ef4f-475b-a326-d3aaf057d256-maas-api-tls" pod "9cbef392-ef4f-475b-a326-d3aaf057d256" (UID: "9cbef392-ef4f-475b-a326-d3aaf057d256"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:21:00.959915 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:00.959894    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cbef392-ef4f-475b-a326-d3aaf057d256-metrics-certs" pod "9cbef392-ef4f-475b-a326-d3aaf057d256" (UID: "9cbef392-ef4f-475b-a326-d3aaf057d256"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:21:00.960181 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:00.960164    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cbef392-ef4f-475b-a326-d3aaf057d256-kube-api-access-frl6x" pod "9cbef392-ef4f-475b-a326-d3aaf057d256" (UID: "9cbef392-ef4f-475b-a326-d3aaf057d256"). InnerVolumeSpecName "kube-api-access-frl6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:21:01.058902 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:01.058811    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cbef392-ef4f-475b-a326-d3aaf057d256-maas-api-tls\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:01.058902 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:01.058847    5053 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cbef392-ef4f-475b-a326-d3aaf057d256-metrics-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:01.058902 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:01.058856    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frl6x\" (UniqueName: \"kubernetes.io/projected/9cbef392-ef4f-475b-a326-d3aaf057d256-kube-api-access-frl6x\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:01.299992 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:01.299958    5053 generic.go:356] "Generic (PLEG): container finished" podID="9cbef392-ef4f-475b-a326-d3aaf057d256" containerID="c8b0df52bec098d47ac04e7d21d96554c6770b69f68e74485eaae9e090bdeaf0" exitCode=0
Nov 03 22:21:01.300191 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:01.300008    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"
Nov 03 22:21:01.300191 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:01.300019    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh" event={"ID":"9cbef392-ef4f-475b-a326-d3aaf057d256","Type":"ContainerDied","Data":"c8b0df52bec098d47ac04e7d21d96554c6770b69f68e74485eaae9e090bdeaf0"}
Nov 03 22:21:01.300191 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:01.300065    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh" event={"ID":"9cbef392-ef4f-475b-a326-d3aaf057d256","Type":"ContainerDied","Data":"233f49a04d16a0699ae9ed12f73aecf8ef2a381b3ca6ba03db78ec2e537f3c95"}
Nov 03 22:21:01.300191 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:01.300087    5053 scope.go:122] "RemoveContainer" containerID="c8b0df52bec098d47ac04e7d21d96554c6770b69f68e74485eaae9e090bdeaf0"
Nov 03 22:21:01.312709 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:01.312686    5053 scope.go:122] "RemoveContainer" containerID="c8b0df52bec098d47ac04e7d21d96554c6770b69f68e74485eaae9e090bdeaf0"
Nov 03 22:21:01.313072 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:21:01.313055    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b0df52bec098d47ac04e7d21d96554c6770b69f68e74485eaae9e090bdeaf0\": container with ID starting with c8b0df52bec098d47ac04e7d21d96554c6770b69f68e74485eaae9e090bdeaf0 not found: ID does not exist" containerID="c8b0df52bec098d47ac04e7d21d96554c6770b69f68e74485eaae9e090bdeaf0"
Nov 03 22:21:01.313132 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:01.313079    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b0df52bec098d47ac04e7d21d96554c6770b69f68e74485eaae9e090bdeaf0"} err="failed to get container status \"c8b0df52bec098d47ac04e7d21d96554c6770b69f68e74485eaae9e090bdeaf0\": rpc error: code = NotFound desc = could not find container \"c8b0df52bec098d47ac04e7d21d96554c6770b69f68e74485eaae9e090bdeaf0\": container with ID starting with c8b0df52bec098d47ac04e7d21d96554c6770b69f68e74485eaae9e090bdeaf0 not found: ID does not exist"
Nov 03 22:21:01.465551 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:01.465516    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"]
Nov 03 22:21:01.468546 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:01.468520    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-534c77-c4964d68b-j62mh"]
Nov 03 22:21:01.667943 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:01.667859    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cbef392-ef4f-475b-a326-d3aaf057d256" path="/var/lib/kubelet/pods/9cbef392-ef4f-475b-a326-d3aaf057d256/volumes"
Nov 03 22:21:09.453657 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:09.453617    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"]
Nov 03 22:21:09.454158 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:09.453876    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689" podUID="886c2db8-998e-4425-a064-630afe2da8ca" containerName="istio-proxy" containerID="cri-o://2842c689efc31dc4c76aa032f86ae8aee5434257e7830eae950b80505b677411" gracePeriod=30
Nov 03 22:21:10.930895 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:21:10.930854    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:21:12.047485 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:12.047449    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"]
Nov 03 22:21:12.047893 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:12.047668    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2" podUID="5b3a23ab-d256-44d9-9dc1-1b4adff44b32" containerName="maas-api" containerID="cri-o://b34986ca34b5448f961e3a587464fa6471b6205ab1e2b9b1f4628f5688143da3" gracePeriod=30
Nov 03 22:21:12.343717 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:12.343688    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"
Nov 03 22:21:12.379476 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:12.379440    5053 generic.go:356] "Generic (PLEG): container finished" podID="5b3a23ab-d256-44d9-9dc1-1b4adff44b32" containerID="b34986ca34b5448f961e3a587464fa6471b6205ab1e2b9b1f4628f5688143da3" exitCode=0
Nov 03 22:21:12.379640 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:12.379494    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"
Nov 03 22:21:12.379640 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:12.379528    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2" event={"ID":"5b3a23ab-d256-44d9-9dc1-1b4adff44b32","Type":"ContainerDied","Data":"b34986ca34b5448f961e3a587464fa6471b6205ab1e2b9b1f4628f5688143da3"}
Nov 03 22:21:12.379640 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:12.379588    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2" event={"ID":"5b3a23ab-d256-44d9-9dc1-1b4adff44b32","Type":"ContainerDied","Data":"d966351927cc67f54ad178a285a5c860dd68ad998575b7266f0f3f89aae58b24"}
Nov 03 22:21:12.379640 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:12.379617    5053 scope.go:122] "RemoveContainer" containerID="b34986ca34b5448f961e3a587464fa6471b6205ab1e2b9b1f4628f5688143da3"
Nov 03 22:21:12.389979 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:12.389957    5053 scope.go:122] "RemoveContainer" containerID="b34986ca34b5448f961e3a587464fa6471b6205ab1e2b9b1f4628f5688143da3"
Nov 03 22:21:12.390322 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:21:12.390293    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34986ca34b5448f961e3a587464fa6471b6205ab1e2b9b1f4628f5688143da3\": container with ID starting with b34986ca34b5448f961e3a587464fa6471b6205ab1e2b9b1f4628f5688143da3 not found: ID does not exist" containerID="b34986ca34b5448f961e3a587464fa6471b6205ab1e2b9b1f4628f5688143da3"
Nov 03 22:21:12.390455 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:12.390333    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34986ca34b5448f961e3a587464fa6471b6205ab1e2b9b1f4628f5688143da3"} err="failed to get container status \"b34986ca34b5448f961e3a587464fa6471b6205ab1e2b9b1f4628f5688143da3\": rpc error: code = NotFound desc = could not find container \"b34986ca34b5448f961e3a587464fa6471b6205ab1e2b9b1f4628f5688143da3\": container with ID starting with b34986ca34b5448f961e3a587464fa6471b6205ab1e2b9b1f4628f5688143da3 not found: ID does not exist"
Nov 03 22:21:12.407452 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:12.407413    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-metrics-certs\") pod \"5b3a23ab-d256-44d9-9dc1-1b4adff44b32\" (UID: \"5b3a23ab-d256-44d9-9dc1-1b4adff44b32\") "
Nov 03 22:21:12.407605 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:12.407541    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-kube-api-access-6rk76\" (UniqueName: \"kubernetes.io/projected/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-kube-api-access-6rk76\") pod \"5b3a23ab-d256-44d9-9dc1-1b4adff44b32\" (UID: \"5b3a23ab-d256-44d9-9dc1-1b4adff44b32\") "
Nov 03 22:21:12.407605 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:12.407598    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-openshift-service-ca\") pod \"5b3a23ab-d256-44d9-9dc1-1b4adff44b32\" (UID: \"5b3a23ab-d256-44d9-9dc1-1b4adff44b32\") "
Nov 03 22:21:12.407743 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:12.407719    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-maas-api-tls\") pod \"5b3a23ab-d256-44d9-9dc1-1b4adff44b32\" (UID: \"5b3a23ab-d256-44d9-9dc1-1b4adff44b32\") "
Nov 03 22:21:12.408013 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:12.407985    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-openshift-service-ca" pod "5b3a23ab-d256-44d9-9dc1-1b4adff44b32" (UID: "5b3a23ab-d256-44d9-9dc1-1b4adff44b32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:21:12.408461 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:12.408183    5053 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-openshift-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:12.410123 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:12.410093    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-kube-api-access-6rk76" pod "5b3a23ab-d256-44d9-9dc1-1b4adff44b32" (UID: "5b3a23ab-d256-44d9-9dc1-1b4adff44b32"). InnerVolumeSpecName "kube-api-access-6rk76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:21:12.410465 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:12.410440    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-metrics-certs" pod "5b3a23ab-d256-44d9-9dc1-1b4adff44b32" (UID: "5b3a23ab-d256-44d9-9dc1-1b4adff44b32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:21:12.410575 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:12.410462    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-maas-api-tls" pod "5b3a23ab-d256-44d9-9dc1-1b4adff44b32" (UID: "5b3a23ab-d256-44d9-9dc1-1b4adff44b32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:21:12.538515 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:12.538475    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rk76\" (UniqueName: \"kubernetes.io/projected/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-kube-api-access-6rk76\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:12.538515 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:12.538515    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-maas-api-tls\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:12.538515 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:12.538531    5053 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b3a23ab-d256-44d9-9dc1-1b4adff44b32-metrics-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:12.801007 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:12.800964    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"]
Nov 03 22:21:12.804380 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:12.804350    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a49a32-67855c7dd7-k8km2"]
Nov 03 22:21:13.667525 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:13.667489    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b3a23ab-d256-44d9-9dc1-1b4adff44b32" path="/var/lib/kubelet/pods/5b3a23ab-d256-44d9-9dc1-1b4adff44b32/volumes"
Nov 03 22:21:14.756716 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.756688    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:21:14.866812 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.866771    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/886c2db8-998e-4425-a064-630afe2da8ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/886c2db8-998e-4425-a064-630afe2da8ca-istiod-ca-cert\") pod \"886c2db8-998e-4425-a064-630afe2da8ca\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") "
Nov 03 22:21:14.867017 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.866859    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/886c2db8-998e-4425-a064-630afe2da8ca-kube-api-access-m98fn\" (UniqueName: \"kubernetes.io/projected/886c2db8-998e-4425-a064-630afe2da8ca-kube-api-access-m98fn\") pod \"886c2db8-998e-4425-a064-630afe2da8ca\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") "
Nov 03 22:21:14.867017 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.866887    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-workload-certs\") pod \"886c2db8-998e-4425-a064-630afe2da8ca\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") "
Nov 03 22:21:14.867017 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.866921    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-workload-socket\") pod \"886c2db8-998e-4425-a064-630afe2da8ca\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") "
Nov 03 22:21:14.867017 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.866956    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/886c2db8-998e-4425-a064-630afe2da8ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/886c2db8-998e-4425-a064-630afe2da8ca-istio-podinfo\") pod \"886c2db8-998e-4425-a064-630afe2da8ca\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") "
Nov 03 22:21:14.867288 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.867066    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/886c2db8-998e-4425-a064-630afe2da8ca-istio-token\" (UniqueName: \"kubernetes.io/projected/886c2db8-998e-4425-a064-630afe2da8ca-istio-token\") pod \"886c2db8-998e-4425-a064-630afe2da8ca\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") "
Nov 03 22:21:14.867288 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.867112    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-istio-envoy\") pod \"886c2db8-998e-4425-a064-630afe2da8ca\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") "
Nov 03 22:21:14.867288 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.867166    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-credential-socket\") pod \"886c2db8-998e-4425-a064-630afe2da8ca\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") "
Nov 03 22:21:14.867288 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.867204    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-workload-socket" pod "886c2db8-998e-4425-a064-630afe2da8ca" (UID: "886c2db8-998e-4425-a064-630afe2da8ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:14.867574 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.867316    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-istio-data\") pod \"886c2db8-998e-4425-a064-630afe2da8ca\" (UID: \"886c2db8-998e-4425-a064-630afe2da8ca\") "
Nov 03 22:21:14.867574 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.867524    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-credential-socket" pod "886c2db8-998e-4425-a064-630afe2da8ca" (UID: "886c2db8-998e-4425-a064-630afe2da8ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:14.867574 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.867556    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/886c2db8-998e-4425-a064-630afe2da8ca-istiod-ca-cert" pod "886c2db8-998e-4425-a064-630afe2da8ca" (UID: "886c2db8-998e-4425-a064-630afe2da8ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:21:14.867750 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.867643    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-istio-data" pod "886c2db8-998e-4425-a064-630afe2da8ca" (UID: "886c2db8-998e-4425-a064-630afe2da8ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:14.867750 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.867728    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-workload-certs" pod "886c2db8-998e-4425-a064-630afe2da8ca" (UID: "886c2db8-998e-4425-a064-630afe2da8ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:14.868049 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.867888    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:14.868049 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.867911    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:14.868049 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.867923    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:14.868049 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.867936    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:14.868049 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.867977    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/886c2db8-998e-4425-a064-630afe2da8ca-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:14.869525 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.869500    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/886c2db8-998e-4425-a064-630afe2da8ca-kube-api-access-m98fn" pod "886c2db8-998e-4425-a064-630afe2da8ca" (UID: "886c2db8-998e-4425-a064-630afe2da8ca"). InnerVolumeSpecName "kube-api-access-m98fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:21:14.869660 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.869619    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/886c2db8-998e-4425-a064-630afe2da8ca-istio-token" pod "886c2db8-998e-4425-a064-630afe2da8ca" (UID: "886c2db8-998e-4425-a064-630afe2da8ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:21:14.869710 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.869691    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-istio-envoy" pod "886c2db8-998e-4425-a064-630afe2da8ca" (UID: "886c2db8-998e-4425-a064-630afe2da8ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:14.869904 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.869882    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/886c2db8-998e-4425-a064-630afe2da8ca-istio-podinfo" pod "886c2db8-998e-4425-a064-630afe2da8ca" (UID: "886c2db8-998e-4425-a064-630afe2da8ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:21:14.968828 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.968781    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m98fn\" (UniqueName: \"kubernetes.io/projected/886c2db8-998e-4425-a064-630afe2da8ca-kube-api-access-m98fn\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:14.968828 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.968817    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/886c2db8-998e-4425-a064-630afe2da8ca-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:14.968828 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.968827    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/886c2db8-998e-4425-a064-630afe2da8ca-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:14.968828 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:14.968836    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/886c2db8-998e-4425-a064-630afe2da8ca-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:15.405322 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:15.405276    5053 generic.go:356] "Generic (PLEG): container finished" podID="886c2db8-998e-4425-a064-630afe2da8ca" containerID="2842c689efc31dc4c76aa032f86ae8aee5434257e7830eae950b80505b677411" exitCode=0
Nov 03 22:21:15.405550 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:15.405346    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"
Nov 03 22:21:15.405550 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:15.405374    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689" event={"ID":"886c2db8-998e-4425-a064-630afe2da8ca","Type":"ContainerDied","Data":"2842c689efc31dc4c76aa032f86ae8aee5434257e7830eae950b80505b677411"}
Nov 03 22:21:15.405550 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:15.405453    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689" event={"ID":"886c2db8-998e-4425-a064-630afe2da8ca","Type":"ContainerDied","Data":"d2a00bcdcf2c5c5d825edaf4ffe207d846525f3fd4369a5a6670c6785f4914b4"}
Nov 03 22:21:15.405550 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:15.405481    5053 scope.go:122] "RemoveContainer" containerID="2842c689efc31dc4c76aa032f86ae8aee5434257e7830eae950b80505b677411"
Nov 03 22:21:15.419030 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:15.419004    5053 scope.go:122] "RemoveContainer" containerID="81b9d9965618291f44b9f9b6b9b57a75adf2913ca0cfd05380f66caaed42325d"
Nov 03 22:21:15.429323 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:15.429289    5053 scope.go:122] "RemoveContainer" containerID="2842c689efc31dc4c76aa032f86ae8aee5434257e7830eae950b80505b677411"
Nov 03 22:21:15.429669 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:21:15.429646    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2842c689efc31dc4c76aa032f86ae8aee5434257e7830eae950b80505b677411\": container with ID starting with 2842c689efc31dc4c76aa032f86ae8aee5434257e7830eae950b80505b677411 not found: ID does not exist" containerID="2842c689efc31dc4c76aa032f86ae8aee5434257e7830eae950b80505b677411"
Nov 03 22:21:15.429778 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:15.429678    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2842c689efc31dc4c76aa032f86ae8aee5434257e7830eae950b80505b677411"} err="failed to get container status \"2842c689efc31dc4c76aa032f86ae8aee5434257e7830eae950b80505b677411\": rpc error: code = NotFound desc = could not find container \"2842c689efc31dc4c76aa032f86ae8aee5434257e7830eae950b80505b677411\": container with ID starting with 2842c689efc31dc4c76aa032f86ae8aee5434257e7830eae950b80505b677411 not found: ID does not exist"
Nov 03 22:21:15.429778 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:15.429699    5053 scope.go:122] "RemoveContainer" containerID="81b9d9965618291f44b9f9b6b9b57a75adf2913ca0cfd05380f66caaed42325d"
Nov 03 22:21:15.430011 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:21:15.429989    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b9d9965618291f44b9f9b6b9b57a75adf2913ca0cfd05380f66caaed42325d\": container with ID starting with 81b9d9965618291f44b9f9b6b9b57a75adf2913ca0cfd05380f66caaed42325d not found: ID does not exist" containerID="81b9d9965618291f44b9f9b6b9b57a75adf2913ca0cfd05380f66caaed42325d"
Nov 03 22:21:15.430064 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:15.430021    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b9d9965618291f44b9f9b6b9b57a75adf2913ca0cfd05380f66caaed42325d"} err="failed to get container status \"81b9d9965618291f44b9f9b6b9b57a75adf2913ca0cfd05380f66caaed42325d\": rpc error: code = NotFound desc = could not find container \"81b9d9965618291f44b9f9b6b9b57a75adf2913ca0cfd05380f66caaed42325d\": container with ID starting with 81b9d9965618291f44b9f9b6b9b57a75adf2913ca0cfd05380f66caaed42325d not found: ID does not exist"
Nov 03 22:21:15.622635 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:15.622596    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"]
Nov 03 22:21:15.629145 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:15.629113    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-534c77-openshift-default-7d66b5b4cc-n2689"]
Nov 03 22:21:15.670276 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:15.670178    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="886c2db8-998e-4425-a064-630afe2da8ca" path="/var/lib/kubelet/pods/886c2db8-998e-4425-a064-630afe2da8ca/volumes"
Nov 03 22:21:23.298192 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:23.298147    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d"]
Nov 03 22:21:23.406456 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:23.406423    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46"]
Nov 03 22:21:28.298864 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:28.298764    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d" podUID="463cdfee-c0ab-4bc1-b51f-20da775292f9" containerName="payload-processing" containerID="cri-o://33c0875f1def25cdd403bb84f56bc77ee254f6d4d74f2d2fcf91223703826795" gracePeriod=25
Nov 03 22:21:28.407769 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:28.407710    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46" podUID="59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d" containerName="payload-pre-processing" containerID="cri-o://6f403b91188771124d71b9c08ec2ed336f0efe383c824e21c94d6c6d533635e4" gracePeriod=25
Nov 03 22:21:28.498214 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:28.498183    5053 generic.go:356] "Generic (PLEG): container finished" podID="463cdfee-c0ab-4bc1-b51f-20da775292f9" containerID="33c0875f1def25cdd403bb84f56bc77ee254f6d4d74f2d2fcf91223703826795" exitCode=0
Nov 03 22:21:28.498450 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:28.498300    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d" event={"ID":"463cdfee-c0ab-4bc1-b51f-20da775292f9","Type":"ContainerDied","Data":"33c0875f1def25cdd403bb84f56bc77ee254f6d4d74f2d2fcf91223703826795"}
Nov 03 22:21:28.590857 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:28.590825    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d"
Nov 03 22:21:28.657395 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:28.657360    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/463cdfee-c0ab-4bc1-b51f-20da775292f9-kube-api-access-zpkdg\" (UniqueName: \"kubernetes.io/projected/463cdfee-c0ab-4bc1-b51f-20da775292f9-kube-api-access-zpkdg\") pod \"463cdfee-c0ab-4bc1-b51f-20da775292f9\" (UID: \"463cdfee-c0ab-4bc1-b51f-20da775292f9\") "
Nov 03 22:21:28.657607 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:28.657444    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/463cdfee-c0ab-4bc1-b51f-20da775292f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/463cdfee-c0ab-4bc1-b51f-20da775292f9-plugins-config-volume\") pod \"463cdfee-c0ab-4bc1-b51f-20da775292f9\" (UID: \"463cdfee-c0ab-4bc1-b51f-20da775292f9\") "
Nov 03 22:21:28.658157 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:28.658130    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/463cdfee-c0ab-4bc1-b51f-20da775292f9-plugins-config-volume" pod "463cdfee-c0ab-4bc1-b51f-20da775292f9" (UID: "463cdfee-c0ab-4bc1-b51f-20da775292f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:21:28.660588 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:28.660562    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/463cdfee-c0ab-4bc1-b51f-20da775292f9-kube-api-access-zpkdg" pod "463cdfee-c0ab-4bc1-b51f-20da775292f9" (UID: "463cdfee-c0ab-4bc1-b51f-20da775292f9"). InnerVolumeSpecName "kube-api-access-zpkdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:21:28.679831 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:28.679808    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46"
Nov 03 22:21:28.758313 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:28.758284    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d-kube-api-access-xjdl7\" (UniqueName: \"kubernetes.io/projected/59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d-kube-api-access-xjdl7\") pod \"59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d\" (UID: \"59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d\") "
Nov 03 22:21:28.758511 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:28.758502    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d-plugins-config-volume\") pod \"59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d\" (UID: \"59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d\") "
Nov 03 22:21:28.758840 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:28.758812    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d-plugins-config-volume" pod "59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d" (UID: "59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:21:28.758996 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:28.758844    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpkdg\" (UniqueName: \"kubernetes.io/projected/463cdfee-c0ab-4bc1-b51f-20da775292f9-kube-api-access-zpkdg\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:28.758996 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:28.758866    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/463cdfee-c0ab-4bc1-b51f-20da775292f9-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:28.760596 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:28.760576    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d-kube-api-access-xjdl7" pod "59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d" (UID: "59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d"). InnerVolumeSpecName "kube-api-access-xjdl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:21:28.860377 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:28.860290    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:28.860377 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:28.860319    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjdl7\" (UniqueName: \"kubernetes.io/projected/59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d-kube-api-access-xjdl7\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:29.509466 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:29.509427    5053 generic.go:356] "Generic (PLEG): container finished" podID="59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d" containerID="6f403b91188771124d71b9c08ec2ed336f0efe383c824e21c94d6c6d533635e4" exitCode=0
Nov 03 22:21:29.509948 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:29.509508    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46"
Nov 03 22:21:29.509948 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:29.509515    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46" event={"ID":"59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d","Type":"ContainerDied","Data":"6f403b91188771124d71b9c08ec2ed336f0efe383c824e21c94d6c6d533635e4"}
Nov 03 22:21:29.509948 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:29.509615    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46" event={"ID":"59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d","Type":"ContainerDied","Data":"75fa67c71757fbe60560b5a7e09913f878c21b87f3eecf48e5d81b1387dc38cf"}
Nov 03 22:21:29.509948 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:29.509647    5053 scope.go:122] "RemoveContainer" containerID="6f403b91188771124d71b9c08ec2ed336f0efe383c824e21c94d6c6d533635e4"
Nov 03 22:21:29.511115 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:29.511098    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d"
Nov 03 22:21:29.511115 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:29.511099    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d" event={"ID":"463cdfee-c0ab-4bc1-b51f-20da775292f9","Type":"ContainerDied","Data":"62c8286361151bae7fc3b2c683a352713827e0aa31d77894a3cef64756a221eb"}
Nov 03 22:21:29.521605 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:29.521533    5053 scope.go:122] "RemoveContainer" containerID="6f403b91188771124d71b9c08ec2ed336f0efe383c824e21c94d6c6d533635e4"
Nov 03 22:21:29.522053 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:21:29.522032    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f403b91188771124d71b9c08ec2ed336f0efe383c824e21c94d6c6d533635e4\": container with ID starting with 6f403b91188771124d71b9c08ec2ed336f0efe383c824e21c94d6c6d533635e4 not found: ID does not exist" containerID="6f403b91188771124d71b9c08ec2ed336f0efe383c824e21c94d6c6d533635e4"
Nov 03 22:21:29.522155 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:29.522061    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f403b91188771124d71b9c08ec2ed336f0efe383c824e21c94d6c6d533635e4"} err="failed to get container status \"6f403b91188771124d71b9c08ec2ed336f0efe383c824e21c94d6c6d533635e4\": rpc error: code = NotFound desc = could not find container \"6f403b91188771124d71b9c08ec2ed336f0efe383c824e21c94d6c6d533635e4\": container with ID starting with 6f403b91188771124d71b9c08ec2ed336f0efe383c824e21c94d6c6d533635e4 not found: ID does not exist"
Nov 03 22:21:29.522155 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:29.522079    5053 scope.go:122] "RemoveContainer" containerID="33c0875f1def25cdd403bb84f56bc77ee254f6d4d74f2d2fcf91223703826795"
Nov 03 22:21:29.674055 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:29.673976    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d"]
Nov 03 22:21:29.676535 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:29.676500    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f463c9-f69d7f4cd-g7l9d"]
Nov 03 22:21:29.784574 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:29.784538    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46"]
Nov 03 22:21:29.788857 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:29.788831    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f463c9-58f7d7df9f-d2h46"]
Nov 03 22:21:31.668010 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:31.667977    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="463cdfee-c0ab-4bc1-b51f-20da775292f9" path="/var/lib/kubelet/pods/463cdfee-c0ab-4bc1-b51f-20da775292f9/volumes"
Nov 03 22:21:31.668453 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:31.668402    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d" path="/var/lib/kubelet/pods/59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d/volumes"
Nov 03 22:21:40.929606 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:21:40.929562    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:21:43.167670 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:43.167633    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"]
Nov 03 22:21:43.168098 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:43.167886    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv" podUID="cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3" containerName="maas-api" containerID="cri-o://6e497ff07b83e80e25bc48d7b673bc4f7daefd95f6a713785d9950f6d573fb1e" gracePeriod=30
Nov 03 22:21:43.445598 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:43.445571    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"
Nov 03 22:21:43.461734 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:43.461692    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-openshift-service-ca\") pod \"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3\" (UID: \"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3\") "
Nov 03 22:21:43.461908 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:43.461865    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-metrics-certs\") pod \"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3\" (UID: \"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3\") "
Nov 03 22:21:43.461908 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:43.461899    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-kube-api-access-m5b2c\" (UniqueName: \"kubernetes.io/projected/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-kube-api-access-m5b2c\") pod \"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3\" (UID: \"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3\") "
Nov 03 22:21:43.462029 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:43.461928    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-maas-api-tls\") pod \"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3\" (UID: \"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3\") "
Nov 03 22:21:43.462143 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:43.462115    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-openshift-service-ca" pod "cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3" (UID: "cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:21:43.462632 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:43.462510    5053 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-openshift-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:43.464228 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:43.464203    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-metrics-certs" pod "cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3" (UID: "cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:21:43.464699 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:43.464679    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-maas-api-tls" pod "cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3" (UID: "cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:21:43.464803 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:43.464784    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-kube-api-access-m5b2c" pod "cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3" (UID: "cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3"). InnerVolumeSpecName "kube-api-access-m5b2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:21:43.563816 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:43.563732    5053 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-metrics-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:43.563816 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:43.563770    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5b2c\" (UniqueName: \"kubernetes.io/projected/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-kube-api-access-m5b2c\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:43.563816 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:43.563785    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3-maas-api-tls\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:43.618440 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:43.618402    5053 generic.go:356] "Generic (PLEG): container finished" podID="cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3" containerID="6e497ff07b83e80e25bc48d7b673bc4f7daefd95f6a713785d9950f6d573fb1e" exitCode=0
Nov 03 22:21:43.618663 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:43.618462    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"
Nov 03 22:21:43.618663 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:43.618485    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv" event={"ID":"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3","Type":"ContainerDied","Data":"6e497ff07b83e80e25bc48d7b673bc4f7daefd95f6a713785d9950f6d573fb1e"}
Nov 03 22:21:43.618663 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:43.618548    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv" event={"ID":"cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3","Type":"ContainerDied","Data":"bb26703227aa566f37255201b5498a1d0d77def3502526d8909c2ea72f0651b3"}
Nov 03 22:21:43.618663 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:43.618577    5053 scope.go:122] "RemoveContainer" containerID="6e497ff07b83e80e25bc48d7b673bc4f7daefd95f6a713785d9950f6d573fb1e"
Nov 03 22:21:43.629527 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:43.629499    5053 scope.go:122] "RemoveContainer" containerID="6e497ff07b83e80e25bc48d7b673bc4f7daefd95f6a713785d9950f6d573fb1e"
Nov 03 22:21:43.629854 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:21:43.629834    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e497ff07b83e80e25bc48d7b673bc4f7daefd95f6a713785d9950f6d573fb1e\": container with ID starting with 6e497ff07b83e80e25bc48d7b673bc4f7daefd95f6a713785d9950f6d573fb1e not found: ID does not exist" containerID="6e497ff07b83e80e25bc48d7b673bc4f7daefd95f6a713785d9950f6d573fb1e"
Nov 03 22:21:43.629909 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:43.629863    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e497ff07b83e80e25bc48d7b673bc4f7daefd95f6a713785d9950f6d573fb1e"} err="failed to get container status \"6e497ff07b83e80e25bc48d7b673bc4f7daefd95f6a713785d9950f6d573fb1e\": rpc error: code = NotFound desc = could not find container \"6e497ff07b83e80e25bc48d7b673bc4f7daefd95f6a713785d9950f6d573fb1e\": container with ID starting with 6e497ff07b83e80e25bc48d7b673bc4f7daefd95f6a713785d9950f6d573fb1e not found: ID does not exist"
Nov 03 22:21:43.782459 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:43.782420    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"]
Nov 03 22:21:43.786131 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:43.786102    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-87e1cf-7c67c4c865-6z5nv"]
Nov 03 22:21:44.463198 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463159    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59888f74-zpnhp"]
Nov 03 22:21:44.463790 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463776    5053 cpu_manager.go:414] "containerMap: removing container" podUID="3d8d06b2-a794-4262-855d-75c8e268f3b5" containerName="istio-proxy"
Nov 03 22:21:44.463844 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463791    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d8d06b2-a794-4262-855d-75c8e268f3b5" containerName="istio-proxy"
Nov 03 22:21:44.463844 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463800    5053 cpu_manager.go:414] "containerMap: removing container" podUID="886c2db8-998e-4425-a064-630afe2da8ca" containerName="istio-proxy"
Nov 03 22:21:44.463844 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463805    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="886c2db8-998e-4425-a064-630afe2da8ca" containerName="istio-proxy"
Nov 03 22:21:44.463844 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463812    5053 cpu_manager.go:414] "containerMap: removing container" podUID="59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d" containerName="payload-pre-processing"
Nov 03 22:21:44.463844 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463817    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d" containerName="payload-pre-processing"
Nov 03 22:21:44.463844 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463828    5053 cpu_manager.go:414] "containerMap: removing container" podUID="982a04b0-d1b7-4f74-ab18-f84ebc337092" containerName="maas-api"
Nov 03 22:21:44.463844 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463833    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="982a04b0-d1b7-4f74-ab18-f84ebc337092" containerName="maas-api"
Nov 03 22:21:44.463844 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463845    5053 cpu_manager.go:414] "containerMap: removing container" podUID="e9b7982f-9598-4fb0-b1d6-10a59db8b02b" containerName="main"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463850    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9b7982f-9598-4fb0-b1d6-10a59db8b02b" containerName="main"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463858    5053 cpu_manager.go:414] "containerMap: removing container" podUID="5b3a23ab-d256-44d9-9dc1-1b4adff44b32" containerName="maas-api"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463863    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b3a23ab-d256-44d9-9dc1-1b4adff44b32" containerName="maas-api"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463870    5053 cpu_manager.go:414] "containerMap: removing container" podUID="4f17d174-9d7f-4b54-82c5-6c4dac27b48b" containerName="istio-proxy"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463874    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f17d174-9d7f-4b54-82c5-6c4dac27b48b" containerName="istio-proxy"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463882    5053 cpu_manager.go:414] "containerMap: removing container" podUID="a6d529fb-027e-4ce2-af6c-50a7f716ddb9" containerName="maas-api"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463886    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6d529fb-027e-4ce2-af6c-50a7f716ddb9" containerName="maas-api"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463895    5053 cpu_manager.go:414] "containerMap: removing container" podUID="4f17d174-9d7f-4b54-82c5-6c4dac27b48b" containerName="istio-proxy"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463899    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f17d174-9d7f-4b54-82c5-6c4dac27b48b" containerName="istio-proxy"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463907    5053 cpu_manager.go:414] "containerMap: removing container" podUID="3d8d06b2-a794-4262-855d-75c8e268f3b5" containerName="istio-proxy"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463911    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d8d06b2-a794-4262-855d-75c8e268f3b5" containerName="istio-proxy"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463919    5053 cpu_manager.go:414] "containerMap: removing container" podUID="722a009c-bbee-44ea-b624-bd8eeb0897b8" containerName="maas-api"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463923    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="722a009c-bbee-44ea-b624-bd8eeb0897b8" containerName="maas-api"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463930    5053 cpu_manager.go:414] "containerMap: removing container" podUID="9cbef392-ef4f-475b-a326-d3aaf057d256" containerName="maas-api"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463935    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cbef392-ef4f-475b-a326-d3aaf057d256" containerName="maas-api"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463945    5053 cpu_manager.go:414] "containerMap: removing container" podUID="463cdfee-c0ab-4bc1-b51f-20da775292f9" containerName="payload-processing"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463949    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="463cdfee-c0ab-4bc1-b51f-20da775292f9" containerName="payload-processing"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463963    5053 cpu_manager.go:414] "containerMap: removing container" podUID="cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3" containerName="maas-api"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.463966    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3" containerName="maas-api"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.464043    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3" containerName="maas-api"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.464051    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="886c2db8-998e-4425-a064-630afe2da8ca" containerName="istio-proxy"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.464058    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d8d06b2-a794-4262-855d-75c8e268f3b5" containerName="istio-proxy"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.464062    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="886c2db8-998e-4425-a064-630afe2da8ca" containerName="istio-proxy"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.464068    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="59f5490d-1a63-49f9-8c8f-3d9a3c4f5c9d" containerName="payload-pre-processing"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.464075    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="982a04b0-d1b7-4f74-ab18-f84ebc337092" containerName="maas-api"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.464082    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9b7982f-9598-4fb0-b1d6-10a59db8b02b" containerName="main"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.464087    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b3a23ab-d256-44d9-9dc1-1b4adff44b32" containerName="maas-api"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.464092    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f17d174-9d7f-4b54-82c5-6c4dac27b48b" containerName="istio-proxy"
Nov 03 22:21:44.464083 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.464098    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6d529fb-027e-4ce2-af6c-50a7f716ddb9" containerName="maas-api"
Nov 03 22:21:44.465037 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.464104    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f17d174-9d7f-4b54-82c5-6c4dac27b48b" containerName="istio-proxy"
Nov 03 22:21:44.465037 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.464109    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d8d06b2-a794-4262-855d-75c8e268f3b5" containerName="istio-proxy"
Nov 03 22:21:44.465037 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.464114    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="722a009c-bbee-44ea-b624-bd8eeb0897b8" containerName="maas-api"
Nov 03 22:21:44.465037 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.464120    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cbef392-ef4f-475b-a326-d3aaf057d256" containerName="maas-api"
Nov 03 22:21:44.465037 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.464125    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="463cdfee-c0ab-4bc1-b51f-20da775292f9" containerName="payload-processing"
Nov 03 22:21:44.469063 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.469035    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59888f74-zpnhp"
Nov 03 22:21:44.525134 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.525087    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59888f74-zpnhp"]
Nov 03 22:21:44.576692 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.576655    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a117058-0ed7-496b-bc07-68dd892f319e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59888f74-zpnhp\" (UID: \"3a117058-0ed7-496b-bc07-68dd892f319e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59888f74-zpnhp"
Nov 03 22:21:44.576876 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.576712    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw5wj\" (UniqueName: \"kubernetes.io/projected/3a117058-0ed7-496b-bc07-68dd892f319e-kube-api-access-lw5wj\") pod \"maas-api-revoke-sub-59888f74-zpnhp\" (UID: \"3a117058-0ed7-496b-bc07-68dd892f319e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59888f74-zpnhp"
Nov 03 22:21:44.576876 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.576840    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a117058-0ed7-496b-bc07-68dd892f319e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59888f74-zpnhp\" (UID: \"3a117058-0ed7-496b-bc07-68dd892f319e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59888f74-zpnhp"
Nov 03 22:21:44.579924 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.579893    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:21:44.580479 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.580457    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:21:44.677575 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.677539    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a117058-0ed7-496b-bc07-68dd892f319e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59888f74-zpnhp\" (UID: \"3a117058-0ed7-496b-bc07-68dd892f319e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59888f74-zpnhp"
Nov 03 22:21:44.677814 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.677602    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw5wj\" (UniqueName: \"kubernetes.io/projected/3a117058-0ed7-496b-bc07-68dd892f319e-kube-api-access-lw5wj\") pod \"maas-api-revoke-sub-59888f74-zpnhp\" (UID: \"3a117058-0ed7-496b-bc07-68dd892f319e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59888f74-zpnhp"
Nov 03 22:21:44.677814 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.677733    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a117058-0ed7-496b-bc07-68dd892f319e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59888f74-zpnhp\" (UID: \"3a117058-0ed7-496b-bc07-68dd892f319e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59888f74-zpnhp"
Nov 03 22:21:44.678628 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.678601    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a117058-0ed7-496b-bc07-68dd892f319e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59888f74-zpnhp\" (UID: \"3a117058-0ed7-496b-bc07-68dd892f319e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59888f74-zpnhp"
Nov 03 22:21:44.680134 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.680107    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a117058-0ed7-496b-bc07-68dd892f319e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59888f74-zpnhp\" (UID: \"3a117058-0ed7-496b-bc07-68dd892f319e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59888f74-zpnhp"
Nov 03 22:21:44.735461 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.735384    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw5wj\" (UniqueName: \"kubernetes.io/projected/3a117058-0ed7-496b-bc07-68dd892f319e-kube-api-access-lw5wj\") pod \"maas-api-revoke-sub-59888f74-zpnhp\" (UID: \"3a117058-0ed7-496b-bc07-68dd892f319e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59888f74-zpnhp"
Nov 03 22:21:44.786966 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:44.786927    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59888f74-zpnhp"
Nov 03 22:21:45.167602 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:45.167576    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59888f74-zpnhp"]
Nov 03 22:21:45.169452 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:21:45.169417    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a117058_0ed7_496b_bc07_68dd892f319e.slice/crio-a3d09b85075d353d569272ae344994a8ad8cb237dbca3cf3ddcfe3a7c8609414 WatchSource:0}: Error finding container a3d09b85075d353d569272ae344994a8ad8cb237dbca3cf3ddcfe3a7c8609414: Status 404 returned error can't find the container with id a3d09b85075d353d569272ae344994a8ad8cb237dbca3cf3ddcfe3a7c8609414
Nov 03 22:21:45.636378 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:45.636346    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59888f74-zpnhp" event={"ID":"3a117058-0ed7-496b-bc07-68dd892f319e","Type":"ContainerStarted","Data":"07c0edfc2aea1e6bea79a86346c73d06d7b7a742f63fd0a7486c9f28894b6272"}
Nov 03 22:21:45.636807 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:45.636399    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59888f74-zpnhp" event={"ID":"3a117058-0ed7-496b-bc07-68dd892f319e","Type":"ContainerStarted","Data":"a3d09b85075d353d569272ae344994a8ad8cb237dbca3cf3ddcfe3a7c8609414"}
Nov 03 22:21:45.667661 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:45.667624    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3" path="/var/lib/kubelet/pods/cc3a1ac9-f959-4e0c-af45-4055ff2ec0d3/volumes"
Nov 03 22:21:45.788610 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:45.788483    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59888f74-zpnhp" podStartSLOduration=1.788462548 podStartE2EDuration="1.788462548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:45.78668469 +0000 UTC m=+4020.280606566" watchObservedRunningTime="2026-11-03 22:21:45.788462548 +0000 UTC m=+4020.282384424"
Nov 03 22:21:46.644306 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:46.644205    5053 generic.go:356] "Generic (PLEG): container finished" podID="3a117058-0ed7-496b-bc07-68dd892f319e" containerID="07c0edfc2aea1e6bea79a86346c73d06d7b7a742f63fd0a7486c9f28894b6272" exitCode=0
Nov 03 22:21:46.644306 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:46.644286    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59888f74-zpnhp" event={"ID":"3a117058-0ed7-496b-bc07-68dd892f319e","Type":"ContainerDied","Data":"07c0edfc2aea1e6bea79a86346c73d06d7b7a742f63fd0a7486c9f28894b6272"}
Nov 03 22:21:47.799113 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:47.799086    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59888f74-zpnhp"
Nov 03 22:21:47.916545 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:47.916472    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a117058-0ed7-496b-bc07-68dd892f319e-kube-api-access-lw5wj\" (UniqueName: \"kubernetes.io/projected/3a117058-0ed7-496b-bc07-68dd892f319e-kube-api-access-lw5wj\") pod \"3a117058-0ed7-496b-bc07-68dd892f319e\" (UID: \"3a117058-0ed7-496b-bc07-68dd892f319e\") "
Nov 03 22:21:47.916709 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:47.916568    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a117058-0ed7-496b-bc07-68dd892f319e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a117058-0ed7-496b-bc07-68dd892f319e-maas-api-openshift-ca\") pod \"3a117058-0ed7-496b-bc07-68dd892f319e\" (UID: \"3a117058-0ed7-496b-bc07-68dd892f319e\") "
Nov 03 22:21:47.916798 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:47.916779    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a117058-0ed7-496b-bc07-68dd892f319e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a117058-0ed7-496b-bc07-68dd892f319e-maas-api-xks-ca\") pod \"3a117058-0ed7-496b-bc07-68dd892f319e\" (UID: \"3a117058-0ed7-496b-bc07-68dd892f319e\") "
Nov 03 22:21:47.917015 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:47.916973    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a117058-0ed7-496b-bc07-68dd892f319e-maas-api-openshift-ca" pod "3a117058-0ed7-496b-bc07-68dd892f319e" (UID: "3a117058-0ed7-496b-bc07-68dd892f319e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:21:47.917354 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:47.917331    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a117058-0ed7-496b-bc07-68dd892f319e-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:47.919552 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:47.919518    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a117058-0ed7-496b-bc07-68dd892f319e-maas-api-xks-ca" pod "3a117058-0ed7-496b-bc07-68dd892f319e" (UID: "3a117058-0ed7-496b-bc07-68dd892f319e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:21:47.919675 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:47.919607    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a117058-0ed7-496b-bc07-68dd892f319e-kube-api-access-lw5wj" pod "3a117058-0ed7-496b-bc07-68dd892f319e" (UID: "3a117058-0ed7-496b-bc07-68dd892f319e"). InnerVolumeSpecName "kube-api-access-lw5wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:21:48.018479 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:48.018446    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw5wj\" (UniqueName: \"kubernetes.io/projected/3a117058-0ed7-496b-bc07-68dd892f319e-kube-api-access-lw5wj\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:48.018479 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:48.018475    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a117058-0ed7-496b-bc07-68dd892f319e-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:48.352502 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:48.352461    5053 scope.go:122] "RemoveContainer" containerID="28a0f94548113edfe9d8c36efebbaf36a386ccb97ee9b80b6640090ab3fda6cb"
Nov 03 22:21:48.660907 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:48.660823    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59888f74-zpnhp"
Nov 03 22:21:48.661095 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:48.660825    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59888f74-zpnhp" event={"ID":"3a117058-0ed7-496b-bc07-68dd892f319e","Type":"ContainerDied","Data":"a3d09b85075d353d569272ae344994a8ad8cb237dbca3cf3ddcfe3a7c8609414"}
Nov 03 22:21:48.661095 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:48.660942    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3d09b85075d353d569272ae344994a8ad8cb237dbca3cf3ddcfe3a7c8609414"
Nov 03 22:21:52.588523 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:52.588487    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"]
Nov 03 22:21:52.588988 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:52.588805    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89" podUID="ccf8ba47-571b-4ad8-ae83-f035c92f485a" containerName="istio-proxy" containerID="cri-o://4da43c0f2ad530ed886d5993a626837cc1c280c140e2546bc159caa9232ad905" gracePeriod=30
Nov 03 22:21:57.722822 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:57.722786    5053 generic.go:356] "Generic (PLEG): container finished" podID="ccf8ba47-571b-4ad8-ae83-f035c92f485a" containerID="4da43c0f2ad530ed886d5993a626837cc1c280c140e2546bc159caa9232ad905" exitCode=0
Nov 03 22:21:57.723309 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:57.722841    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89" event={"ID":"ccf8ba47-571b-4ad8-ae83-f035c92f485a","Type":"ContainerDied","Data":"4da43c0f2ad530ed886d5993a626837cc1c280c140e2546bc159caa9232ad905"}
Nov 03 22:21:57.907036 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:57.907003    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 22:21:58.069383 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.069350    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-token\" (UniqueName: \"kubernetes.io/projected/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-token\") pod \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") "
Nov 03 22:21:58.069612 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.069396    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-envoy\") pod \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") "
Nov 03 22:21:58.069612 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.069427    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf8ba47-571b-4ad8-ae83-f035c92f485a-kube-api-access-hhdvg\" (UniqueName: \"kubernetes.io/projected/ccf8ba47-571b-4ad8-ae83-f035c92f485a-kube-api-access-hhdvg\") pod \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") "
Nov 03 22:21:58.069612 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.069454    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-workload-certs\") pod \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") "
Nov 03 22:21:58.069612 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.069513    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-data\") pod \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") "
Nov 03 22:21:58.069612 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.069543    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-credential-socket\") pod \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") "
Nov 03 22:21:58.069612 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.069568    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-workload-socket\") pod \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") "
Nov 03 22:21:58.069967 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.069633    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-podinfo\") pod \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") "
Nov 03 22:21:58.069967 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.069693    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istiod-ca-cert\") pod \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\" (UID: \"ccf8ba47-571b-4ad8-ae83-f035c92f485a\") "
Nov 03 22:21:58.069967 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.069786    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-workload-certs" pod "ccf8ba47-571b-4ad8-ae83-f035c92f485a" (UID: "ccf8ba47-571b-4ad8-ae83-f035c92f485a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:58.069967 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.069837    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-data" pod "ccf8ba47-571b-4ad8-ae83-f035c92f485a" (UID: "ccf8ba47-571b-4ad8-ae83-f035c92f485a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:58.070193 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.069997    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-workload-socket" pod "ccf8ba47-571b-4ad8-ae83-f035c92f485a" (UID: "ccf8ba47-571b-4ad8-ae83-f035c92f485a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:58.070193 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.070079    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-credential-socket" pod "ccf8ba47-571b-4ad8-ae83-f035c92f485a" (UID: "ccf8ba47-571b-4ad8-ae83-f035c92f485a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:58.070363 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.070326    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istiod-ca-cert" pod "ccf8ba47-571b-4ad8-ae83-f035c92f485a" (UID: "ccf8ba47-571b-4ad8-ae83-f035c92f485a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:21:58.070571 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.070519    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:58.070571 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.070548    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:58.070571 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.070565    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:58.070571 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.070578    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:58.070571 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.070594    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:58.072390 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.072141    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf8ba47-571b-4ad8-ae83-f035c92f485a-kube-api-access-hhdvg" pod "ccf8ba47-571b-4ad8-ae83-f035c92f485a" (UID: "ccf8ba47-571b-4ad8-ae83-f035c92f485a"). InnerVolumeSpecName "kube-api-access-hhdvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:21:58.072390 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.072170    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-podinfo" pod "ccf8ba47-571b-4ad8-ae83-f035c92f485a" (UID: "ccf8ba47-571b-4ad8-ae83-f035c92f485a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:21:58.072689 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.072662    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-envoy" pod "ccf8ba47-571b-4ad8-ae83-f035c92f485a" (UID: "ccf8ba47-571b-4ad8-ae83-f035c92f485a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:58.072815 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.072798    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-token" pod "ccf8ba47-571b-4ad8-ae83-f035c92f485a" (UID: "ccf8ba47-571b-4ad8-ae83-f035c92f485a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:21:58.172297 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.172180    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:58.172297 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.172228    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:58.172297 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.172276    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf8ba47-571b-4ad8-ae83-f035c92f485a-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:58.172297 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.172286    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhdvg\" (UniqueName: \"kubernetes.io/projected/ccf8ba47-571b-4ad8-ae83-f035c92f485a-kube-api-access-hhdvg\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:58.733201 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.733170    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"
Nov 03 22:21:58.733201 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.733187    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89" event={"ID":"ccf8ba47-571b-4ad8-ae83-f035c92f485a","Type":"ContainerDied","Data":"142759b9bc924a15e9b9e8f26aba1878b4119730700b921ebed01f07c2a0f4f9"}
Nov 03 22:21:58.733784 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.733289    5053 scope.go:122] "RemoveContainer" containerID="4da43c0f2ad530ed886d5993a626837cc1c280c140e2546bc159caa9232ad905"
Nov 03 22:21:58.952760 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:58.952723    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"]
Nov 03 22:21:59.002010 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:59.001922    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-zbr89"]
Nov 03 22:21:59.667185 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:21:59.667152    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccf8ba47-571b-4ad8-ae83-f035c92f485a" path="/var/lib/kubelet/pods/ccf8ba47-571b-4ad8-ae83-f035c92f485a/volumes"
Nov 03 22:22:01.070780 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:01.070740    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"]
Nov 03 22:22:01.071204 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:01.070949    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv" podUID="bef7697e-1c56-48a7-9758-3243cd62e8f1" containerName="maas-api" containerID="cri-o://a5232b3ad663f517c8b08eddca19b9353d745adf7868452652f6e940cabab80a" gracePeriod=30
Nov 03 22:22:01.351898 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:01.351868    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"
Nov 03 22:22:01.512451 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:01.512415    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bef7697e-1c56-48a7-9758-3243cd62e8f1-kube-api-access-n5s9f\" (UniqueName: \"kubernetes.io/projected/bef7697e-1c56-48a7-9758-3243cd62e8f1-kube-api-access-n5s9f\") pod \"bef7697e-1c56-48a7-9758-3243cd62e8f1\" (UID: \"bef7697e-1c56-48a7-9758-3243cd62e8f1\") "
Nov 03 22:22:01.512664 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:01.512492    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bef7697e-1c56-48a7-9758-3243cd62e8f1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bef7697e-1c56-48a7-9758-3243cd62e8f1-metrics-certs\") pod \"bef7697e-1c56-48a7-9758-3243cd62e8f1\" (UID: \"bef7697e-1c56-48a7-9758-3243cd62e8f1\") "
Nov 03 22:22:01.512664 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:01.512614    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bef7697e-1c56-48a7-9758-3243cd62e8f1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bef7697e-1c56-48a7-9758-3243cd62e8f1-maas-api-tls\") pod \"bef7697e-1c56-48a7-9758-3243cd62e8f1\" (UID: \"bef7697e-1c56-48a7-9758-3243cd62e8f1\") "
Nov 03 22:22:01.512664 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:01.512656    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bef7697e-1c56-48a7-9758-3243cd62e8f1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef7697e-1c56-48a7-9758-3243cd62e8f1-openshift-service-ca\") pod \"bef7697e-1c56-48a7-9758-3243cd62e8f1\" (UID: \"bef7697e-1c56-48a7-9758-3243cd62e8f1\") "
Nov 03 22:22:01.513085 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:01.513053    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef7697e-1c56-48a7-9758-3243cd62e8f1-openshift-service-ca" pod "bef7697e-1c56-48a7-9758-3243cd62e8f1" (UID: "bef7697e-1c56-48a7-9758-3243cd62e8f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:22:01.515065 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:01.515041    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef7697e-1c56-48a7-9758-3243cd62e8f1-kube-api-access-n5s9f" pod "bef7697e-1c56-48a7-9758-3243cd62e8f1" (UID: "bef7697e-1c56-48a7-9758-3243cd62e8f1"). InnerVolumeSpecName "kube-api-access-n5s9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:01.515191 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:01.515172    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef7697e-1c56-48a7-9758-3243cd62e8f1-metrics-certs" pod "bef7697e-1c56-48a7-9758-3243cd62e8f1" (UID: "bef7697e-1c56-48a7-9758-3243cd62e8f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:22:01.515336 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:01.515311    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef7697e-1c56-48a7-9758-3243cd62e8f1-maas-api-tls" pod "bef7697e-1c56-48a7-9758-3243cd62e8f1" (UID: "bef7697e-1c56-48a7-9758-3243cd62e8f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:22:01.614709 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:01.614611    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bef7697e-1c56-48a7-9758-3243cd62e8f1-maas-api-tls\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:01.614709 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:01.614644    5053 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef7697e-1c56-48a7-9758-3243cd62e8f1-openshift-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:01.614709 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:01.614654    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5s9f\" (UniqueName: \"kubernetes.io/projected/bef7697e-1c56-48a7-9758-3243cd62e8f1-kube-api-access-n5s9f\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:01.614709 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:01.614664    5053 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bef7697e-1c56-48a7-9758-3243cd62e8f1-metrics-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:01.755779 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:01.755742    5053 generic.go:356] "Generic (PLEG): container finished" podID="bef7697e-1c56-48a7-9758-3243cd62e8f1" containerID="a5232b3ad663f517c8b08eddca19b9353d745adf7868452652f6e940cabab80a" exitCode=0
Nov 03 22:22:01.755994 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:01.755810    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"
Nov 03 22:22:01.755994 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:01.755827    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv" event={"ID":"bef7697e-1c56-48a7-9758-3243cd62e8f1","Type":"ContainerDied","Data":"a5232b3ad663f517c8b08eddca19b9353d745adf7868452652f6e940cabab80a"}
Nov 03 22:22:01.755994 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:01.755881    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv" event={"ID":"bef7697e-1c56-48a7-9758-3243cd62e8f1","Type":"ContainerDied","Data":"87de27ba7b54c033200147704433e6dfcce45d97972d601038a98916bb9785ad"}
Nov 03 22:22:01.755994 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:01.755905    5053 scope.go:122] "RemoveContainer" containerID="a5232b3ad663f517c8b08eddca19b9353d745adf7868452652f6e940cabab80a"
Nov 03 22:22:01.766552 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:01.766530    5053 scope.go:122] "RemoveContainer" containerID="a5232b3ad663f517c8b08eddca19b9353d745adf7868452652f6e940cabab80a"
Nov 03 22:22:01.766888 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:22:01.766867    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5232b3ad663f517c8b08eddca19b9353d745adf7868452652f6e940cabab80a\": container with ID starting with a5232b3ad663f517c8b08eddca19b9353d745adf7868452652f6e940cabab80a not found: ID does not exist" containerID="a5232b3ad663f517c8b08eddca19b9353d745adf7868452652f6e940cabab80a"
Nov 03 22:22:01.766961 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:01.766898    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5232b3ad663f517c8b08eddca19b9353d745adf7868452652f6e940cabab80a"} err="failed to get container status \"a5232b3ad663f517c8b08eddca19b9353d745adf7868452652f6e940cabab80a\": rpc error: code = NotFound desc = could not find container \"a5232b3ad663f517c8b08eddca19b9353d745adf7868452652f6e940cabab80a\": container with ID starting with a5232b3ad663f517c8b08eddca19b9353d745adf7868452652f6e940cabab80a not found: ID does not exist"
Nov 03 22:22:01.918703 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:01.918616    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"]
Nov 03 22:22:01.924321 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:01.924292    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0d45ba-5659c47786-nlrrv"]
Nov 03 22:22:03.666690 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:03.666653    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bef7697e-1c56-48a7-9758-3243cd62e8f1" path="/var/lib/kubelet/pods/bef7697e-1c56-48a7-9758-3243cd62e8f1/volumes"
Nov 03 22:22:06.154072 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.154038    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l"]
Nov 03 22:22:06.154684 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.154659    5053 cpu_manager.go:414] "containerMap: removing container" podUID="3a117058-0ed7-496b-bc07-68dd892f319e" containerName="revoke-keys"
Nov 03 22:22:06.154684 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.154684    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a117058-0ed7-496b-bc07-68dd892f319e" containerName="revoke-keys"
Nov 03 22:22:06.154858 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.154716    5053 cpu_manager.go:414] "containerMap: removing container" podUID="bef7697e-1c56-48a7-9758-3243cd62e8f1" containerName="maas-api"
Nov 03 22:22:06.154858 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.154725    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef7697e-1c56-48a7-9758-3243cd62e8f1" containerName="maas-api"
Nov 03 22:22:06.154858 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.154767    5053 cpu_manager.go:414] "containerMap: removing container" podUID="ccf8ba47-571b-4ad8-ae83-f035c92f485a" containerName="istio-proxy"
Nov 03 22:22:06.154858 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.154772    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf8ba47-571b-4ad8-ae83-f035c92f485a" containerName="istio-proxy"
Nov 03 22:22:06.154858 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.154778    5053 cpu_manager.go:414] "containerMap: removing container" podUID="886c2db8-998e-4425-a064-630afe2da8ca" containerName="istio-proxy"
Nov 03 22:22:06.154858 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.154782    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="886c2db8-998e-4425-a064-630afe2da8ca" containerName="istio-proxy"
Nov 03 22:22:06.155047 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.154874    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf8ba47-571b-4ad8-ae83-f035c92f485a" containerName="istio-proxy"
Nov 03 22:22:06.155047 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.154884    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a117058-0ed7-496b-bc07-68dd892f319e" containerName="revoke-keys"
Nov 03 22:22:06.155047 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.154899    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef7697e-1c56-48a7-9758-3243cd62e8f1" containerName="maas-api"
Nov 03 22:22:06.159819 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.159803    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l"
Nov 03 22:22:06.212125 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.212093    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:22:06.212408 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.212109    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:22:06.214126 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.214097    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l"]
Nov 03 22:22:06.270828 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.270792    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab33391e-7f3d-4b28-a61b-a311284df2ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l\" (UID: \"ab33391e-7f3d-4b28-a61b-a311284df2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l"
Nov 03 22:22:06.271057 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.270982    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab33391e-7f3d-4b28-a61b-a311284df2ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l\" (UID: \"ab33391e-7f3d-4b28-a61b-a311284df2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l"
Nov 03 22:22:06.271057 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.271050    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb7pk\" (UniqueName: \"kubernetes.io/projected/ab33391e-7f3d-4b28-a61b-a311284df2ab-kube-api-access-bb7pk\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l\" (UID: \"ab33391e-7f3d-4b28-a61b-a311284df2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l"
Nov 03 22:22:06.372131 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.372083    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab33391e-7f3d-4b28-a61b-a311284df2ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l\" (UID: \"ab33391e-7f3d-4b28-a61b-a311284df2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l"
Nov 03 22:22:06.377322 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.377231    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab33391e-7f3d-4b28-a61b-a311284df2ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l\" (UID: \"ab33391e-7f3d-4b28-a61b-a311284df2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l"
Nov 03 22:22:06.396541 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.395371    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb7pk\" (UniqueName: \"kubernetes.io/projected/ab33391e-7f3d-4b28-a61b-a311284df2ab-kube-api-access-bb7pk\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l\" (UID: \"ab33391e-7f3d-4b28-a61b-a311284df2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l"
Nov 03 22:22:06.396541 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.395536    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab33391e-7f3d-4b28-a61b-a311284df2ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l\" (UID: \"ab33391e-7f3d-4b28-a61b-a311284df2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l"
Nov 03 22:22:06.396541 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.396484    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab33391e-7f3d-4b28-a61b-a311284df2ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l\" (UID: \"ab33391e-7f3d-4b28-a61b-a311284df2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l"
Nov 03 22:22:06.432465 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.432394    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb7pk\" (UniqueName: \"kubernetes.io/projected/ab33391e-7f3d-4b28-a61b-a311284df2ab-kube-api-access-bb7pk\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l\" (UID: \"ab33391e-7f3d-4b28-a61b-a311284df2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l"
Nov 03 22:22:06.473433 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.473401    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l"
Nov 03 22:22:06.851165 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:06.851137    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l"]
Nov 03 22:22:06.852972 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:22:06.852938    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab33391e_7f3d_4b28_a61b_a311284df2ab.slice/crio-349ecc653e15a188fed495bc12b7d8e208ab59ed86a44868c672964d6d6db95a WatchSource:0}: Error finding container 349ecc653e15a188fed495bc12b7d8e208ab59ed86a44868c672964d6d6db95a: Status 404 returned error can't find the container with id 349ecc653e15a188fed495bc12b7d8e208ab59ed86a44868c672964d6d6db95a
Nov 03 22:22:07.796334 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:07.796297    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l" event={"ID":"ab33391e-7f3d-4b28-a61b-a311284df2ab","Type":"ContainerStarted","Data":"268b3c9f36c664e746432afab225d06a362d1c3d4f77e392e0e9a1b00f754660"}
Nov 03 22:22:07.796801 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:07.796352    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l" event={"ID":"ab33391e-7f3d-4b28-a61b-a311284df2ab","Type":"ContainerStarted","Data":"349ecc653e15a188fed495bc12b7d8e208ab59ed86a44868c672964d6d6db95a"}
Nov 03 22:22:07.911569 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:07.911508    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l" podStartSLOduration=1.91149198 podStartE2EDuration="1.91149198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:07.9087252 +0000 UTC m=+4042.402647075" watchObservedRunningTime="2026-11-03 22:22:07.91149198 +0000 UTC m=+4042.405413909"
Nov 03 22:22:08.804334 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:08.804285    5053 generic.go:356] "Generic (PLEG): container finished" podID="ab33391e-7f3d-4b28-a61b-a311284df2ab" containerID="268b3c9f36c664e746432afab225d06a362d1c3d4f77e392e0e9a1b00f754660" exitCode=0
Nov 03 22:22:08.804760 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:08.804383    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l" event={"ID":"ab33391e-7f3d-4b28-a61b-a311284df2ab","Type":"ContainerDied","Data":"268b3c9f36c664e746432afab225d06a362d1c3d4f77e392e0e9a1b00f754660"}
Nov 03 22:22:09.971028 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:09.971003    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l"
Nov 03 22:22:10.050489 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:10.050449    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab33391e-7f3d-4b28-a61b-a311284df2ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab33391e-7f3d-4b28-a61b-a311284df2ab-maas-api-openshift-ca\") pod \"ab33391e-7f3d-4b28-a61b-a311284df2ab\" (UID: \"ab33391e-7f3d-4b28-a61b-a311284df2ab\") "
Nov 03 22:22:10.050690 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:10.050628    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab33391e-7f3d-4b28-a61b-a311284df2ab-kube-api-access-bb7pk\" (UniqueName: \"kubernetes.io/projected/ab33391e-7f3d-4b28-a61b-a311284df2ab-kube-api-access-bb7pk\") pod \"ab33391e-7f3d-4b28-a61b-a311284df2ab\" (UID: \"ab33391e-7f3d-4b28-a61b-a311284df2ab\") "
Nov 03 22:22:10.050768 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:10.050704    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab33391e-7f3d-4b28-a61b-a311284df2ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab33391e-7f3d-4b28-a61b-a311284df2ab-maas-api-xks-ca\") pod \"ab33391e-7f3d-4b28-a61b-a311284df2ab\" (UID: \"ab33391e-7f3d-4b28-a61b-a311284df2ab\") "
Nov 03 22:22:10.050887 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:10.050859    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab33391e-7f3d-4b28-a61b-a311284df2ab-maas-api-openshift-ca" pod "ab33391e-7f3d-4b28-a61b-a311284df2ab" (UID: "ab33391e-7f3d-4b28-a61b-a311284df2ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:22:10.051127 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:10.051109    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab33391e-7f3d-4b28-a61b-a311284df2ab-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:10.054160 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:10.054133    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab33391e-7f3d-4b28-a61b-a311284df2ab-kube-api-access-bb7pk" pod "ab33391e-7f3d-4b28-a61b-a311284df2ab" (UID: "ab33391e-7f3d-4b28-a61b-a311284df2ab"). InnerVolumeSpecName "kube-api-access-bb7pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:10.055603 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:10.055572    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab33391e-7f3d-4b28-a61b-a311284df2ab-maas-api-xks-ca" pod "ab33391e-7f3d-4b28-a61b-a311284df2ab" (UID: "ab33391e-7f3d-4b28-a61b-a311284df2ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:22:10.075598 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:10.075524    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"]
Nov 03 22:22:10.075764 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:10.075742    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf" podUID="0c981726-d50f-4694-93d4-ebefb45bfb47" containerName="istio-proxy" containerID="cri-o://c35dc0340a1c39f0945da7cd44caa61e92c613acae1761c117c9c45278b8a5dc" gracePeriod=30
Nov 03 22:22:10.153945 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:10.153903    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb7pk\" (UniqueName: \"kubernetes.io/projected/ab33391e-7f3d-4b28-a61b-a311284df2ab-kube-api-access-bb7pk\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:10.153945 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:10.153947    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab33391e-7f3d-4b28-a61b-a311284df2ab-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:10.822208 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:10.822176    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l"
Nov 03 22:22:10.822438 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:10.822210    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l" event={"ID":"ab33391e-7f3d-4b28-a61b-a311284df2ab","Type":"ContainerDied","Data":"349ecc653e15a188fed495bc12b7d8e208ab59ed86a44868c672964d6d6db95a"}
Nov 03 22:22:10.822438 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:10.822296    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="349ecc653e15a188fed495bc12b7d8e208ab59ed86a44868c672964d6d6db95a"
Nov 03 22:22:10.931738 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:22:10.931697    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:22:11.012383 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:11.012346    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn"]
Nov 03 22:22:11.012896 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:11.012881    5053 cpu_manager.go:414] "containerMap: removing container" podUID="ab33391e-7f3d-4b28-a61b-a311284df2ab" containerName="revoke-keys"
Nov 03 22:22:11.012965 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:11.012896    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab33391e-7f3d-4b28-a61b-a311284df2ab" containerName="revoke-keys"
Nov 03 22:22:11.013025 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:11.013000    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab33391e-7f3d-4b28-a61b-a311284df2ab" containerName="revoke-keys"
Nov 03 22:22:11.016457 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:11.016435    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn"
Nov 03 22:22:11.069283 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:11.069224    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:22:11.069481 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:11.069388    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:22:11.072916 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:11.072834    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn"]
Nov 03 22:22:11.165168 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:11.165133    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4201e070-91d5-445d-8eff-57bc5ef9ed00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn\" (UID: \"4201e070-91d5-445d-8eff-57bc5ef9ed00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn"
Nov 03 22:22:11.165390 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:11.165175    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4201e070-91d5-445d-8eff-57bc5ef9ed00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn\" (UID: \"4201e070-91d5-445d-8eff-57bc5ef9ed00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn"
Nov 03 22:22:11.165479 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:11.165456    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwxfm\" (UniqueName: \"kubernetes.io/projected/4201e070-91d5-445d-8eff-57bc5ef9ed00-kube-api-access-dwxfm\") pod \"maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn\" (UID: \"4201e070-91d5-445d-8eff-57bc5ef9ed00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn"
Nov 03 22:22:11.266646 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:11.266610    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwxfm\" (UniqueName: \"kubernetes.io/projected/4201e070-91d5-445d-8eff-57bc5ef9ed00-kube-api-access-dwxfm\") pod \"maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn\" (UID: \"4201e070-91d5-445d-8eff-57bc5ef9ed00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn"
Nov 03 22:22:11.266888 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:11.266665    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4201e070-91d5-445d-8eff-57bc5ef9ed00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn\" (UID: \"4201e070-91d5-445d-8eff-57bc5ef9ed00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn"
Nov 03 22:22:11.266888 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:11.266705    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4201e070-91d5-445d-8eff-57bc5ef9ed00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn\" (UID: \"4201e070-91d5-445d-8eff-57bc5ef9ed00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn"
Nov 03 22:22:11.267482 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:11.267455    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4201e070-91d5-445d-8eff-57bc5ef9ed00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn\" (UID: \"4201e070-91d5-445d-8eff-57bc5ef9ed00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn"
Nov 03 22:22:11.269338 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:11.269316    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4201e070-91d5-445d-8eff-57bc5ef9ed00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn\" (UID: \"4201e070-91d5-445d-8eff-57bc5ef9ed00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn"
Nov 03 22:22:11.323302 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:11.323209    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwxfm\" (UniqueName: \"kubernetes.io/projected/4201e070-91d5-445d-8eff-57bc5ef9ed00-kube-api-access-dwxfm\") pod \"maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn\" (UID: \"4201e070-91d5-445d-8eff-57bc5ef9ed00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn"
Nov 03 22:22:11.330846 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:11.330817    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn"
Nov 03 22:22:11.706344 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:11.706315    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn"]
Nov 03 22:22:11.709623 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:22:11.709588    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4201e070_91d5_445d_8eff_57bc5ef9ed00.slice/crio-e00f542b1ead7f10f650bb614ec0dc8cda7605c22755766273287aaabc073cc7 WatchSource:0}: Error finding container e00f542b1ead7f10f650bb614ec0dc8cda7605c22755766273287aaabc073cc7: Status 404 returned error can't find the container with id e00f542b1ead7f10f650bb614ec0dc8cda7605c22755766273287aaabc073cc7
Nov 03 22:22:11.833330 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:11.833274    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn" event={"ID":"4201e070-91d5-445d-8eff-57bc5ef9ed00","Type":"ContainerStarted","Data":"e00f542b1ead7f10f650bb614ec0dc8cda7605c22755766273287aaabc073cc7"}
Nov 03 22:22:12.737521 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:12.737478    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"]
Nov 03 22:22:12.737994 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:12.737676    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb" podUID="47fec592-cd7c-4208-92e9-c8dc38e1a7f1" containerName="maas-api" containerID="cri-o://78e6e4194eabfb434203f347007c2159b87bfff2b3a9ffeab6726b386a462c2c" gracePeriod=30
Nov 03 22:22:12.845050 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:12.844951    5053 generic.go:356] "Generic (PLEG): container finished" podID="4201e070-91d5-445d-8eff-57bc5ef9ed00" containerID="731248d4d6d24a59a98dc7815377949479afd8cdaf3fc9ac32ce08a041944008" exitCode=0
Nov 03 22:22:12.845223 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:12.845040    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn" event={"ID":"4201e070-91d5-445d-8eff-57bc5ef9ed00","Type":"ContainerDied","Data":"731248d4d6d24a59a98dc7815377949479afd8cdaf3fc9ac32ce08a041944008"}
Nov 03 22:22:13.011989 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:13.011962    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"
Nov 03 22:22:13.090034 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:13.089987    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-kube-api-access-88v7w\" (UniqueName: \"kubernetes.io/projected/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-kube-api-access-88v7w\") pod \"47fec592-cd7c-4208-92e9-c8dc38e1a7f1\" (UID: \"47fec592-cd7c-4208-92e9-c8dc38e1a7f1\") "
Nov 03 22:22:13.090266 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:13.090061    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-openshift-service-ca\") pod \"47fec592-cd7c-4208-92e9-c8dc38e1a7f1\" (UID: \"47fec592-cd7c-4208-92e9-c8dc38e1a7f1\") "
Nov 03 22:22:13.090266 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:13.090191    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-metrics-certs\") pod \"47fec592-cd7c-4208-92e9-c8dc38e1a7f1\" (UID: \"47fec592-cd7c-4208-92e9-c8dc38e1a7f1\") "
Nov 03 22:22:13.090266 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:13.090224    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-maas-api-tls\") pod \"47fec592-cd7c-4208-92e9-c8dc38e1a7f1\" (UID: \"47fec592-cd7c-4208-92e9-c8dc38e1a7f1\") "
Nov 03 22:22:13.090472 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:13.090442    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-openshift-service-ca" pod "47fec592-cd7c-4208-92e9-c8dc38e1a7f1" (UID: "47fec592-cd7c-4208-92e9-c8dc38e1a7f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:22:13.090732 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:13.090713    5053 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-openshift-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:13.092492 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:13.092469    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-metrics-certs" pod "47fec592-cd7c-4208-92e9-c8dc38e1a7f1" (UID: "47fec592-cd7c-4208-92e9-c8dc38e1a7f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:22:13.092492 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:13.092487    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-maas-api-tls" pod "47fec592-cd7c-4208-92e9-c8dc38e1a7f1" (UID: "47fec592-cd7c-4208-92e9-c8dc38e1a7f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:22:13.092639 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:13.092498    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-kube-api-access-88v7w" pod "47fec592-cd7c-4208-92e9-c8dc38e1a7f1" (UID: "47fec592-cd7c-4208-92e9-c8dc38e1a7f1"). InnerVolumeSpecName "kube-api-access-88v7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:13.192201 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:13.192109    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88v7w\" (UniqueName: \"kubernetes.io/projected/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-kube-api-access-88v7w\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:13.192201 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:13.192143    5053 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-metrics-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:13.192201 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:13.192153    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47fec592-cd7c-4208-92e9-c8dc38e1a7f1-maas-api-tls\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:13.852391 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:13.852355    5053 generic.go:356] "Generic (PLEG): container finished" podID="47fec592-cd7c-4208-92e9-c8dc38e1a7f1" containerID="78e6e4194eabfb434203f347007c2159b87bfff2b3a9ffeab6726b386a462c2c" exitCode=0
Nov 03 22:22:13.852862 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:13.852415    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"
Nov 03 22:22:13.852862 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:13.852437    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb" event={"ID":"47fec592-cd7c-4208-92e9-c8dc38e1a7f1","Type":"ContainerDied","Data":"78e6e4194eabfb434203f347007c2159b87bfff2b3a9ffeab6726b386a462c2c"}
Nov 03 22:22:13.852862 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:13.852490    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb" event={"ID":"47fec592-cd7c-4208-92e9-c8dc38e1a7f1","Type":"ContainerDied","Data":"fd9e59962f0423efb12dac98d4cf9ea87de109fb9389480c837f54b23096fd3a"}
Nov 03 22:22:13.852862 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:13.852521    5053 scope.go:122] "RemoveContainer" containerID="78e6e4194eabfb434203f347007c2159b87bfff2b3a9ffeab6726b386a462c2c"
Nov 03 22:22:13.862385 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:13.862362    5053 scope.go:122] "RemoveContainer" containerID="78e6e4194eabfb434203f347007c2159b87bfff2b3a9ffeab6726b386a462c2c"
Nov 03 22:22:13.862736 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:22:13.862710    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78e6e4194eabfb434203f347007c2159b87bfff2b3a9ffeab6726b386a462c2c\": container with ID starting with 78e6e4194eabfb434203f347007c2159b87bfff2b3a9ffeab6726b386a462c2c not found: ID does not exist" containerID="78e6e4194eabfb434203f347007c2159b87bfff2b3a9ffeab6726b386a462c2c"
Nov 03 22:22:13.862851 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:13.862741    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e6e4194eabfb434203f347007c2159b87bfff2b3a9ffeab6726b386a462c2c"} err="failed to get container status \"78e6e4194eabfb434203f347007c2159b87bfff2b3a9ffeab6726b386a462c2c\": rpc error: code = NotFound desc = could not find container \"78e6e4194eabfb434203f347007c2159b87bfff2b3a9ffeab6726b386a462c2c\": container with ID starting with 78e6e4194eabfb434203f347007c2159b87bfff2b3a9ffeab6726b386a462c2c not found: ID does not exist"
Nov 03 22:22:14.013735 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:14.013706    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn"
Nov 03 22:22:14.015509 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:14.015487    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"]
Nov 03 22:22:14.065349 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:14.065305    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75f577-bc7f8f467-chxpb"]
Nov 03 22:22:14.103660 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:14.103625    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4201e070-91d5-445d-8eff-57bc5ef9ed00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4201e070-91d5-445d-8eff-57bc5ef9ed00-maas-api-xks-ca\") pod \"4201e070-91d5-445d-8eff-57bc5ef9ed00\" (UID: \"4201e070-91d5-445d-8eff-57bc5ef9ed00\") "
Nov 03 22:22:14.103834 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:14.103696    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4201e070-91d5-445d-8eff-57bc5ef9ed00-kube-api-access-dwxfm\" (UniqueName: \"kubernetes.io/projected/4201e070-91d5-445d-8eff-57bc5ef9ed00-kube-api-access-dwxfm\") pod \"4201e070-91d5-445d-8eff-57bc5ef9ed00\" (UID: \"4201e070-91d5-445d-8eff-57bc5ef9ed00\") "
Nov 03 22:22:14.103834 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:14.103807    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4201e070-91d5-445d-8eff-57bc5ef9ed00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4201e070-91d5-445d-8eff-57bc5ef9ed00-maas-api-openshift-ca\") pod \"4201e070-91d5-445d-8eff-57bc5ef9ed00\" (UID: \"4201e070-91d5-445d-8eff-57bc5ef9ed00\") "
Nov 03 22:22:14.104205 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:14.104175    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4201e070-91d5-445d-8eff-57bc5ef9ed00-maas-api-openshift-ca" pod "4201e070-91d5-445d-8eff-57bc5ef9ed00" (UID: "4201e070-91d5-445d-8eff-57bc5ef9ed00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:22:14.105797 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:14.105776    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4201e070-91d5-445d-8eff-57bc5ef9ed00-maas-api-xks-ca" pod "4201e070-91d5-445d-8eff-57bc5ef9ed00" (UID: "4201e070-91d5-445d-8eff-57bc5ef9ed00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:22:14.105942 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:14.105923    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4201e070-91d5-445d-8eff-57bc5ef9ed00-kube-api-access-dwxfm" pod "4201e070-91d5-445d-8eff-57bc5ef9ed00" (UID: "4201e070-91d5-445d-8eff-57bc5ef9ed00"). InnerVolumeSpecName "kube-api-access-dwxfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:14.205609 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:14.205569    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4201e070-91d5-445d-8eff-57bc5ef9ed00-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:14.205609 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:14.205603    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4201e070-91d5-445d-8eff-57bc5ef9ed00-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:14.205609 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:14.205614    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwxfm\" (UniqueName: \"kubernetes.io/projected/4201e070-91d5-445d-8eff-57bc5ef9ed00-kube-api-access-dwxfm\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:14.860788 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:14.860757    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn"
Nov 03 22:22:14.861217 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:14.860802    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn" event={"ID":"4201e070-91d5-445d-8eff-57bc5ef9ed00","Type":"ContainerDied","Data":"e00f542b1ead7f10f650bb614ec0dc8cda7605c22755766273287aaabc073cc7"}
Nov 03 22:22:14.861217 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:14.860853    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e00f542b1ead7f10f650bb614ec0dc8cda7605c22755766273287aaabc073cc7"
Nov 03 22:22:15.334033 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.334007    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:22:15.420576 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.420543    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-envoy\") pod \"0c981726-d50f-4694-93d4-ebefb45bfb47\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") "
Nov 03 22:22:15.420576 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.420592    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-token\" (UniqueName: \"kubernetes.io/projected/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-token\") pod \"0c981726-d50f-4694-93d4-ebefb45bfb47\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") "
Nov 03 22:22:15.420835 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.420649    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c981726-d50f-4694-93d4-ebefb45bfb47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c981726-d50f-4694-93d4-ebefb45bfb47-istiod-ca-cert\") pod \"0c981726-d50f-4694-93d4-ebefb45bfb47\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") "
Nov 03 22:22:15.420835 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.420690    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-credential-socket\") pod \"0c981726-d50f-4694-93d4-ebefb45bfb47\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") "
Nov 03 22:22:15.420835 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.420715    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-data\") pod \"0c981726-d50f-4694-93d4-ebefb45bfb47\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") "
Nov 03 22:22:15.420835 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.420770    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-podinfo\") pod \"0c981726-d50f-4694-93d4-ebefb45bfb47\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") "
Nov 03 22:22:15.420835 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.420822    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-workload-certs\") pod \"0c981726-d50f-4694-93d4-ebefb45bfb47\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") "
Nov 03 22:22:15.421116 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.420850    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c981726-d50f-4694-93d4-ebefb45bfb47-kube-api-access-lprhz\" (UniqueName: \"kubernetes.io/projected/0c981726-d50f-4694-93d4-ebefb45bfb47-kube-api-access-lprhz\") pod \"0c981726-d50f-4694-93d4-ebefb45bfb47\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") "
Nov 03 22:22:15.421116 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.420892    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-workload-socket\") pod \"0c981726-d50f-4694-93d4-ebefb45bfb47\" (UID: \"0c981726-d50f-4694-93d4-ebefb45bfb47\") "
Nov 03 22:22:15.421116 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.421021    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-credential-socket" pod "0c981726-d50f-4694-93d4-ebefb45bfb47" (UID: "0c981726-d50f-4694-93d4-ebefb45bfb47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:15.421334 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.421195    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-workload-certs" pod "0c981726-d50f-4694-93d4-ebefb45bfb47" (UID: "0c981726-d50f-4694-93d4-ebefb45bfb47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:15.421334 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.421205    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c981726-d50f-4694-93d4-ebefb45bfb47-istiod-ca-cert" pod "0c981726-d50f-4694-93d4-ebefb45bfb47" (UID: "0c981726-d50f-4694-93d4-ebefb45bfb47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:22:15.421480 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.421454    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-workload-socket" pod "0c981726-d50f-4694-93d4-ebefb45bfb47" (UID: "0c981726-d50f-4694-93d4-ebefb45bfb47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:15.421543 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.421484    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-data" pod "0c981726-d50f-4694-93d4-ebefb45bfb47" (UID: "0c981726-d50f-4694-93d4-ebefb45bfb47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:15.421692 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.421671    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c981726-d50f-4694-93d4-ebefb45bfb47-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:15.421769 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.421699    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:15.421769 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.421729    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:15.423115 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.423092    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-token" pod "0c981726-d50f-4694-93d4-ebefb45bfb47" (UID: "0c981726-d50f-4694-93d4-ebefb45bfb47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:15.423229 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.423153    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-envoy" pod "0c981726-d50f-4694-93d4-ebefb45bfb47" (UID: "0c981726-d50f-4694-93d4-ebefb45bfb47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:15.423229 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.423173    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c981726-d50f-4694-93d4-ebefb45bfb47-kube-api-access-lprhz" pod "0c981726-d50f-4694-93d4-ebefb45bfb47" (UID: "0c981726-d50f-4694-93d4-ebefb45bfb47"). InnerVolumeSpecName "kube-api-access-lprhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:15.423413 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.423398    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-podinfo" pod "0c981726-d50f-4694-93d4-ebefb45bfb47" (UID: "0c981726-d50f-4694-93d4-ebefb45bfb47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:22:15.507896 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.507849    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"]
Nov 03 22:22:15.513135 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.513095    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"]
Nov 03 22:22:15.522729 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.522693    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"]
Nov 03 22:22:15.523015 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.522981    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lprhz\" (UniqueName: \"kubernetes.io/projected/0c981726-d50f-4694-93d4-ebefb45bfb47-kube-api-access-lprhz\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:15.523063 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.523013    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:15.523063 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.523028    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:15.523063 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.523040    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:15.523063 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.523054    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:15.523226 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.523066    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c981726-d50f-4694-93d4-ebefb45bfb47-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:15.531080 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.531009    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"]
Nov 03 22:22:15.666780 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.666741    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47fec592-cd7c-4208-92e9-c8dc38e1a7f1" path="/var/lib/kubelet/pods/47fec592-cd7c-4208-92e9-c8dc38e1a7f1/volumes"
Nov 03 22:22:15.869435 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.869351    5053 generic.go:356] "Generic (PLEG): container finished" podID="0c981726-d50f-4694-93d4-ebefb45bfb47" containerID="c35dc0340a1c39f0945da7cd44caa61e92c613acae1761c117c9c45278b8a5dc" exitCode=0
Nov 03 22:22:15.869435 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.869405    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"
Nov 03 22:22:15.869966 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.869441    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf" event={"ID":"0c981726-d50f-4694-93d4-ebefb45bfb47","Type":"ContainerDied","Data":"c35dc0340a1c39f0945da7cd44caa61e92c613acae1761c117c9c45278b8a5dc"}
Nov 03 22:22:15.869966 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.869506    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf" event={"ID":"0c981726-d50f-4694-93d4-ebefb45bfb47","Type":"ContainerDied","Data":"997db12c76c76a0d1e6c74d4ea73bf937181f7e09b3ce37f1acbacb5c10f6919"}
Nov 03 22:22:15.869966 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.869531    5053 scope.go:122] "RemoveContainer" containerID="c35dc0340a1c39f0945da7cd44caa61e92c613acae1761c117c9c45278b8a5dc"
Nov 03 22:22:15.879695 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.879674    5053 scope.go:122] "RemoveContainer" containerID="0c20c4d038d8d2aba7c4324a914c750bb2fd8aa17bcea5c98b7140aafe24eb51"
Nov 03 22:22:15.889443 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.889422    5053 scope.go:122] "RemoveContainer" containerID="c35dc0340a1c39f0945da7cd44caa61e92c613acae1761c117c9c45278b8a5dc"
Nov 03 22:22:15.889742 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:22:15.889725    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c35dc0340a1c39f0945da7cd44caa61e92c613acae1761c117c9c45278b8a5dc\": container with ID starting with c35dc0340a1c39f0945da7cd44caa61e92c613acae1761c117c9c45278b8a5dc not found: ID does not exist" containerID="c35dc0340a1c39f0945da7cd44caa61e92c613acae1761c117c9c45278b8a5dc"
Nov 03 22:22:15.889787 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.889749    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c35dc0340a1c39f0945da7cd44caa61e92c613acae1761c117c9c45278b8a5dc"} err="failed to get container status \"c35dc0340a1c39f0945da7cd44caa61e92c613acae1761c117c9c45278b8a5dc\": rpc error: code = NotFound desc = could not find container \"c35dc0340a1c39f0945da7cd44caa61e92c613acae1761c117c9c45278b8a5dc\": container with ID starting with c35dc0340a1c39f0945da7cd44caa61e92c613acae1761c117c9c45278b8a5dc not found: ID does not exist"
Nov 03 22:22:15.889787 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.889767    5053 scope.go:122] "RemoveContainer" containerID="0c20c4d038d8d2aba7c4324a914c750bb2fd8aa17bcea5c98b7140aafe24eb51"
Nov 03 22:22:15.890042 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:22:15.890028    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c20c4d038d8d2aba7c4324a914c750bb2fd8aa17bcea5c98b7140aafe24eb51\": container with ID starting with 0c20c4d038d8d2aba7c4324a914c750bb2fd8aa17bcea5c98b7140aafe24eb51 not found: ID does not exist" containerID="0c20c4d038d8d2aba7c4324a914c750bb2fd8aa17bcea5c98b7140aafe24eb51"
Nov 03 22:22:15.890111 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:15.890044    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c20c4d038d8d2aba7c4324a914c750bb2fd8aa17bcea5c98b7140aafe24eb51"} err="failed to get container status \"0c20c4d038d8d2aba7c4324a914c750bb2fd8aa17bcea5c98b7140aafe24eb51\": rpc error: code = NotFound desc = could not find container \"0c20c4d038d8d2aba7c4324a914c750bb2fd8aa17bcea5c98b7140aafe24eb51\": container with ID starting with 0c20c4d038d8d2aba7c4324a914c750bb2fd8aa17bcea5c98b7140aafe24eb51 not found: ID does not exist"
Nov 03 22:22:16.126637 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:16.126552    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"]
Nov 03 22:22:16.130389 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:16.130358    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-0d45ba-openshift-default-7f7fd6794f-pf6qf"]
Nov 03 22:22:17.667304 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:17.667260    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c981726-d50f-4694-93d4-ebefb45bfb47" path="/var/lib/kubelet/pods/0c981726-d50f-4694-93d4-ebefb45bfb47/volumes"
Nov 03 22:22:22.371080 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.371043    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h"]
Nov 03 22:22:22.371724 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.371707    5053 cpu_manager.go:414] "containerMap: removing container" podUID="0c981726-d50f-4694-93d4-ebefb45bfb47" containerName="istio-proxy"
Nov 03 22:22:22.371724 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.371725    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c981726-d50f-4694-93d4-ebefb45bfb47" containerName="istio-proxy"
Nov 03 22:22:22.371852 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.371735    5053 cpu_manager.go:414] "containerMap: removing container" podUID="4201e070-91d5-445d-8eff-57bc5ef9ed00" containerName="revoke-keys"
Nov 03 22:22:22.371852 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.371741    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4201e070-91d5-445d-8eff-57bc5ef9ed00" containerName="revoke-keys"
Nov 03 22:22:22.371852 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.371760    5053 cpu_manager.go:414] "containerMap: removing container" podUID="0c981726-d50f-4694-93d4-ebefb45bfb47" containerName="istio-proxy"
Nov 03 22:22:22.371852 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.371766    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c981726-d50f-4694-93d4-ebefb45bfb47" containerName="istio-proxy"
Nov 03 22:22:22.371852 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.371790    5053 cpu_manager.go:414] "containerMap: removing container" podUID="47fec592-cd7c-4208-92e9-c8dc38e1a7f1" containerName="maas-api"
Nov 03 22:22:22.371852 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.371795    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47fec592-cd7c-4208-92e9-c8dc38e1a7f1" containerName="maas-api"
Nov 03 22:22:22.372293 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.371884    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="47fec592-cd7c-4208-92e9-c8dc38e1a7f1" containerName="maas-api"
Nov 03 22:22:22.372293 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.371898    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c981726-d50f-4694-93d4-ebefb45bfb47" containerName="istio-proxy"
Nov 03 22:22:22.372293 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.371905    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="4201e070-91d5-445d-8eff-57bc5ef9ed00" containerName="revoke-keys"
Nov 03 22:22:22.372293 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.371914    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c981726-d50f-4694-93d4-ebefb45bfb47" containerName="istio-proxy"
Nov 03 22:22:22.375576 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.375551    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h"
Nov 03 22:22:22.400974 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.400940    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ad58fc-198d-4b28-9f7a-380f56bc6257-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h\" (UID: \"f4ad58fc-198d-4b28-9f7a-380f56bc6257\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h"
Nov 03 22:22:22.401219 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.401150    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxq4w\" (UniqueName: \"kubernetes.io/projected/f4ad58fc-198d-4b28-9f7a-380f56bc6257-kube-api-access-fxq4w\") pod \"maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h\" (UID: \"f4ad58fc-198d-4b28-9f7a-380f56bc6257\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h"
Nov 03 22:22:22.401361 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.401330    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ad58fc-198d-4b28-9f7a-380f56bc6257-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h\" (UID: \"f4ad58fc-198d-4b28-9f7a-380f56bc6257\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h"
Nov 03 22:22:22.443927 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.443888    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h"]
Nov 03 22:22:22.484544 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.484505    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:22:22.484729 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.484685    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:22:22.502062 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.502015    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ad58fc-198d-4b28-9f7a-380f56bc6257-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h\" (UID: \"f4ad58fc-198d-4b28-9f7a-380f56bc6257\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h"
Nov 03 22:22:22.502282 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.502116    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxq4w\" (UniqueName: \"kubernetes.io/projected/f4ad58fc-198d-4b28-9f7a-380f56bc6257-kube-api-access-fxq4w\") pod \"maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h\" (UID: \"f4ad58fc-198d-4b28-9f7a-380f56bc6257\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h"
Nov 03 22:22:22.502282 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.502156    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ad58fc-198d-4b28-9f7a-380f56bc6257-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h\" (UID: \"f4ad58fc-198d-4b28-9f7a-380f56bc6257\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h"
Nov 03 22:22:22.502852 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.502826    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ad58fc-198d-4b28-9f7a-380f56bc6257-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h\" (UID: \"f4ad58fc-198d-4b28-9f7a-380f56bc6257\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h"
Nov 03 22:22:22.504747 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.504720    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ad58fc-198d-4b28-9f7a-380f56bc6257-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h\" (UID: \"f4ad58fc-198d-4b28-9f7a-380f56bc6257\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h"
Nov 03 22:22:22.559123 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.559087    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxq4w\" (UniqueName: \"kubernetes.io/projected/f4ad58fc-198d-4b28-9f7a-380f56bc6257-kube-api-access-fxq4w\") pod \"maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h\" (UID: \"f4ad58fc-198d-4b28-9f7a-380f56bc6257\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h"
Nov 03 22:22:22.695015 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:22.694983    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h"
Nov 03 22:22:23.075999 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:23.075957    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h"]
Nov 03 22:22:23.077425 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:22:23.077399    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ad58fc_198d_4b28_9f7a_380f56bc6257.slice/crio-c5c6331a3f2bdd0c2884edca13331a7dd857c97758a5922efb13811fff9100c0 WatchSource:0}: Error finding container c5c6331a3f2bdd0c2884edca13331a7dd857c97758a5922efb13811fff9100c0: Status 404 returned error can't find the container with id c5c6331a3f2bdd0c2884edca13331a7dd857c97758a5922efb13811fff9100c0
Nov 03 22:22:23.759475 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:23.759434    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b"]
Nov 03 22:22:23.868752 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:23.868721    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9"]
Nov 03 22:22:23.919587 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:23.919545    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h" event={"ID":"f4ad58fc-198d-4b28-9f7a-380f56bc6257","Type":"ContainerStarted","Data":"53892c5d8cb23167e2a188e5be4ab8c6e322a2e365ae35bd339bcab3eb5d55fc"}
Nov 03 22:22:23.919773 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:23.919615    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h" event={"ID":"f4ad58fc-198d-4b28-9f7a-380f56bc6257","Type":"ContainerStarted","Data":"c5c6331a3f2bdd0c2884edca13331a7dd857c97758a5922efb13811fff9100c0"}
Nov 03 22:22:24.102724 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:24.102671    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h" podStartSLOduration=2.102655821 podStartE2EDuration="2.102655821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:24.100750249 +0000 UTC m=+4058.594672124" watchObservedRunningTime="2026-11-03 22:22:24.102655821 +0000 UTC m=+4058.596577703"
Nov 03 22:22:24.926565 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:24.926523    5053 generic.go:356] "Generic (PLEG): container finished" podID="f4ad58fc-198d-4b28-9f7a-380f56bc6257" containerID="53892c5d8cb23167e2a188e5be4ab8c6e322a2e365ae35bd339bcab3eb5d55fc" exitCode=0
Nov 03 22:22:24.926981 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:24.926606    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h" event={"ID":"f4ad58fc-198d-4b28-9f7a-380f56bc6257","Type":"ContainerDied","Data":"53892c5d8cb23167e2a188e5be4ab8c6e322a2e365ae35bd339bcab3eb5d55fc"}
Nov 03 22:22:26.090878 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:26.090852    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h"
Nov 03 22:22:26.158849 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:26.158809    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4ad58fc-198d-4b28-9f7a-380f56bc6257-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ad58fc-198d-4b28-9f7a-380f56bc6257-maas-api-openshift-ca\") pod \"f4ad58fc-198d-4b28-9f7a-380f56bc6257\" (UID: \"f4ad58fc-198d-4b28-9f7a-380f56bc6257\") "
Nov 03 22:22:26.159061 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:26.158909    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4ad58fc-198d-4b28-9f7a-380f56bc6257-kube-api-access-fxq4w\" (UniqueName: \"kubernetes.io/projected/f4ad58fc-198d-4b28-9f7a-380f56bc6257-kube-api-access-fxq4w\") pod \"f4ad58fc-198d-4b28-9f7a-380f56bc6257\" (UID: \"f4ad58fc-198d-4b28-9f7a-380f56bc6257\") "
Nov 03 22:22:26.159061 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:26.158937    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4ad58fc-198d-4b28-9f7a-380f56bc6257-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ad58fc-198d-4b28-9f7a-380f56bc6257-maas-api-xks-ca\") pod \"f4ad58fc-198d-4b28-9f7a-380f56bc6257\" (UID: \"f4ad58fc-198d-4b28-9f7a-380f56bc6257\") "
Nov 03 22:22:26.159211 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:26.159186    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ad58fc-198d-4b28-9f7a-380f56bc6257-maas-api-openshift-ca" pod "f4ad58fc-198d-4b28-9f7a-380f56bc6257" (UID: "f4ad58fc-198d-4b28-9f7a-380f56bc6257"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:22:26.159507 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:26.159480    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ad58fc-198d-4b28-9f7a-380f56bc6257-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:26.161217 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:26.161189    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4ad58fc-198d-4b28-9f7a-380f56bc6257-maas-api-xks-ca" pod "f4ad58fc-198d-4b28-9f7a-380f56bc6257" (UID: "f4ad58fc-198d-4b28-9f7a-380f56bc6257"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:22:26.161359 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:26.161340    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ad58fc-198d-4b28-9f7a-380f56bc6257-kube-api-access-fxq4w" pod "f4ad58fc-198d-4b28-9f7a-380f56bc6257" (UID: "f4ad58fc-198d-4b28-9f7a-380f56bc6257"). InnerVolumeSpecName "kube-api-access-fxq4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:26.260878 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:26.260841    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxq4w\" (UniqueName: \"kubernetes.io/projected/f4ad58fc-198d-4b28-9f7a-380f56bc6257-kube-api-access-fxq4w\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:26.260878 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:26.260876    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ad58fc-198d-4b28-9f7a-380f56bc6257-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:26.939748 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:26.939716    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h"
Nov 03 22:22:26.939940 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:26.939753    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h" event={"ID":"f4ad58fc-198d-4b28-9f7a-380f56bc6257","Type":"ContainerDied","Data":"c5c6331a3f2bdd0c2884edca13331a7dd857c97758a5922efb13811fff9100c0"}
Nov 03 22:22:26.939940 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:26.939798    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5c6331a3f2bdd0c2884edca13331a7dd857c97758a5922efb13811fff9100c0"
Nov 03 22:22:28.760324 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:28.760269    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b" podUID="b4036e26-b7be-4c14-bd68-4863da3c3fe0" containerName="payload-processing" containerID="cri-o://dff7de5b02b5d5a738a6e8e311a95694d4761c80d6a740c6a35354ac743898b2" gracePeriod=25
Nov 03 22:22:28.869511 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:28.869460    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9" podUID="4cee4995-fb03-43bc-a63d-f764dba58a39" containerName="payload-pre-processing" containerID="cri-o://cd8dc383b439f79e13e223c48e54410069c13ff033541735d99b9980018e9fda" gracePeriod=25
Nov 03 22:22:30.530402 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.530352    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6" podUID="4803807f-9312-40ac-badb-00c00d3f3bd3" containerName="main" containerID="cri-o://57791de81b25257bb1d5c63321a87dce0bf674c5fc02a0ea5fe43ed6f0a5c8a5" gracePeriod=45
Nov 03 22:22:30.534395 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.534357    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8" podUID="43404f1c-c87c-4325-ab97-bffb6ea81e57" containerName="main" containerID="cri-o://05761c88f26f13861c837676cdcec7871393c6846daddbe2d7fa5993e9b8a78e" gracePeriod=45
Nov 03 22:22:30.545223 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.545181    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx" podUID="a6c0d3aa-18c2-465b-846b-b82968acc780" containerName="main" containerID="cri-o://d2f61c28ec45b9976a0984df736177a90e6da2f254aa92fcd5e0887a20f26136" gracePeriod=45
Nov 03 22:22:30.551678 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.551647    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n" podUID="512b3ab1-45ba-4f96-9799-b129b25fca42" containerName="main" containerID="cri-o://ea8eff9b3ce3e0af6dc6411f546ca09e4d2d311835ef7b61935bab7cf41944d5" gracePeriod=45
Nov 03 22:22:30.641252 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.641189    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx" podUID="a6c0d3aa-18c2-465b-846b-b82968acc780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": dial tcp 10.132.0.80:8000: connect: connection refused"
Nov 03 22:22:30.732675 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.720771    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n" podUID="512b3ab1-45ba-4f96-9799-b129b25fca42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": dial tcp 10.132.0.79:8000: connect: connection refused"
Nov 03 22:22:30.880000 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.879671    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 22:22:30.897609 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.897575    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 22:22:30.914735 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.914655    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 22:22:30.921450 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.921423    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 22:22:30.967445 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.967393    5053 generic.go:356] "Generic (PLEG): container finished" podID="a6c0d3aa-18c2-465b-846b-b82968acc780" containerID="d2f61c28ec45b9976a0984df736177a90e6da2f254aa92fcd5e0887a20f26136" exitCode=0
Nov 03 22:22:30.967445 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.967440    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"
Nov 03 22:22:30.967728 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.967458    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx" event={"ID":"a6c0d3aa-18c2-465b-846b-b82968acc780","Type":"ContainerDied","Data":"d2f61c28ec45b9976a0984df736177a90e6da2f254aa92fcd5e0887a20f26136"}
Nov 03 22:22:30.967728 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.967524    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx" event={"ID":"a6c0d3aa-18c2-465b-846b-b82968acc780","Type":"ContainerDied","Data":"8b16c1530c78751acfd6ce77276711475e0010363a56d42cb96f6c94819a2678"}
Nov 03 22:22:30.967728 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.967545    5053 scope.go:122] "RemoveContainer" containerID="d2f61c28ec45b9976a0984df736177a90e6da2f254aa92fcd5e0887a20f26136"
Nov 03 22:22:30.969563 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.969540    5053 generic.go:356] "Generic (PLEG): container finished" podID="512b3ab1-45ba-4f96-9799-b129b25fca42" containerID="ea8eff9b3ce3e0af6dc6411f546ca09e4d2d311835ef7b61935bab7cf41944d5" exitCode=0
Nov 03 22:22:30.969679 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.969605    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"
Nov 03 22:22:30.969679 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.969609    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n" event={"ID":"512b3ab1-45ba-4f96-9799-b129b25fca42","Type":"ContainerDied","Data":"ea8eff9b3ce3e0af6dc6411f546ca09e4d2d311835ef7b61935bab7cf41944d5"}
Nov 03 22:22:30.969679 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.969645    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n" event={"ID":"512b3ab1-45ba-4f96-9799-b129b25fca42","Type":"ContainerDied","Data":"fff1e89092d2d489c3c33c26190dda886bd63fbef34c04c6bc80560732500556"}
Nov 03 22:22:30.971134 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.971103    5053 generic.go:356] "Generic (PLEG): container finished" podID="4803807f-9312-40ac-badb-00c00d3f3bd3" containerID="57791de81b25257bb1d5c63321a87dce0bf674c5fc02a0ea5fe43ed6f0a5c8a5" exitCode=0
Nov 03 22:22:30.971282 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.971138    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"
Nov 03 22:22:30.971282 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.971181    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6" event={"ID":"4803807f-9312-40ac-badb-00c00d3f3bd3","Type":"ContainerDied","Data":"57791de81b25257bb1d5c63321a87dce0bf674c5fc02a0ea5fe43ed6f0a5c8a5"}
Nov 03 22:22:30.971282 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.971212    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6" event={"ID":"4803807f-9312-40ac-badb-00c00d3f3bd3","Type":"ContainerDied","Data":"1352a7cde22407f8365e63dfa12cd6a98886dca53d57411a1540132f31c4e13a"}
Nov 03 22:22:30.972635 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.972615    5053 generic.go:356] "Generic (PLEG): container finished" podID="43404f1c-c87c-4325-ab97-bffb6ea81e57" containerID="05761c88f26f13861c837676cdcec7871393c6846daddbe2d7fa5993e9b8a78e" exitCode=0
Nov 03 22:22:30.972732 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.972646    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"
Nov 03 22:22:30.972732 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.972675    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8" event={"ID":"43404f1c-c87c-4325-ab97-bffb6ea81e57","Type":"ContainerDied","Data":"05761c88f26f13861c837676cdcec7871393c6846daddbe2d7fa5993e9b8a78e"}
Nov 03 22:22:30.972732 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.972698    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8" event={"ID":"43404f1c-c87c-4325-ab97-bffb6ea81e57","Type":"ContainerDied","Data":"d4e6a5ad2e1e435f0b2e28bdc0219228fd2a48e1599e47a6b6b15abd3746e750"}
Nov 03 22:22:30.977575 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.977552    5053 scope.go:122] "RemoveContainer" containerID="d2f61c28ec45b9976a0984df736177a90e6da2f254aa92fcd5e0887a20f26136"
Nov 03 22:22:30.977844 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:22:30.977827    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2f61c28ec45b9976a0984df736177a90e6da2f254aa92fcd5e0887a20f26136\": container with ID starting with d2f61c28ec45b9976a0984df736177a90e6da2f254aa92fcd5e0887a20f26136 not found: ID does not exist" containerID="d2f61c28ec45b9976a0984df736177a90e6da2f254aa92fcd5e0887a20f26136"
Nov 03 22:22:30.977889 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.977852    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f61c28ec45b9976a0984df736177a90e6da2f254aa92fcd5e0887a20f26136"} err="failed to get container status \"d2f61c28ec45b9976a0984df736177a90e6da2f254aa92fcd5e0887a20f26136\": rpc error: code = NotFound desc = could not find container \"d2f61c28ec45b9976a0984df736177a90e6da2f254aa92fcd5e0887a20f26136\": container with ID starting with d2f61c28ec45b9976a0984df736177a90e6da2f254aa92fcd5e0887a20f26136 not found: ID does not exist"
Nov 03 22:22:30.977889 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.977870    5053 scope.go:122] "RemoveContainer" containerID="ea8eff9b3ce3e0af6dc6411f546ca09e4d2d311835ef7b61935bab7cf41944d5"
Nov 03 22:22:30.996551 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.996345    5053 scope.go:122] "RemoveContainer" containerID="ea8eff9b3ce3e0af6dc6411f546ca09e4d2d311835ef7b61935bab7cf41944d5"
Nov 03 22:22:30.996812 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:22:30.996788    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea8eff9b3ce3e0af6dc6411f546ca09e4d2d311835ef7b61935bab7cf41944d5\": container with ID starting with ea8eff9b3ce3e0af6dc6411f546ca09e4d2d311835ef7b61935bab7cf41944d5 not found: ID does not exist" containerID="ea8eff9b3ce3e0af6dc6411f546ca09e4d2d311835ef7b61935bab7cf41944d5"
Nov 03 22:22:30.996932 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.996824    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8eff9b3ce3e0af6dc6411f546ca09e4d2d311835ef7b61935bab7cf41944d5"} err="failed to get container status \"ea8eff9b3ce3e0af6dc6411f546ca09e4d2d311835ef7b61935bab7cf41944d5\": rpc error: code = NotFound desc = could not find container \"ea8eff9b3ce3e0af6dc6411f546ca09e4d2d311835ef7b61935bab7cf41944d5\": container with ID starting with ea8eff9b3ce3e0af6dc6411f546ca09e4d2d311835ef7b61935bab7cf41944d5 not found: ID does not exist"
Nov 03 22:22:30.996932 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:30.996854    5053 scope.go:122] "RemoveContainer" containerID="57791de81b25257bb1d5c63321a87dce0bf674c5fc02a0ea5fe43ed6f0a5c8a5"
Nov 03 22:22:31.011477 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.009599    5053 scope.go:122] "RemoveContainer" containerID="57791de81b25257bb1d5c63321a87dce0bf674c5fc02a0ea5fe43ed6f0a5c8a5"
Nov 03 22:22:31.011477 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:22:31.011420    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57791de81b25257bb1d5c63321a87dce0bf674c5fc02a0ea5fe43ed6f0a5c8a5\": container with ID starting with 57791de81b25257bb1d5c63321a87dce0bf674c5fc02a0ea5fe43ed6f0a5c8a5 not found: ID does not exist" containerID="57791de81b25257bb1d5c63321a87dce0bf674c5fc02a0ea5fe43ed6f0a5c8a5"
Nov 03 22:22:31.012120 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.011462    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57791de81b25257bb1d5c63321a87dce0bf674c5fc02a0ea5fe43ed6f0a5c8a5"} err="failed to get container status \"57791de81b25257bb1d5c63321a87dce0bf674c5fc02a0ea5fe43ed6f0a5c8a5\": rpc error: code = NotFound desc = could not find container \"57791de81b25257bb1d5c63321a87dce0bf674c5fc02a0ea5fe43ed6f0a5c8a5\": container with ID starting with 57791de81b25257bb1d5c63321a87dce0bf674c5fc02a0ea5fe43ed6f0a5c8a5 not found: ID does not exist"
Nov 03 22:22:31.012120 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.011642    5053 scope.go:122] "RemoveContainer" containerID="05761c88f26f13861c837676cdcec7871393c6846daddbe2d7fa5993e9b8a78e"
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.019839    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6c0d3aa-18c2-465b-846b-b82968acc780-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c0d3aa-18c2-465b-846b-b82968acc780-tls-certs\") pod \"a6c0d3aa-18c2-465b-846b-b82968acc780\" (UID: \"a6c0d3aa-18c2-465b-846b-b82968acc780\") "
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.019912    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-tmp-dir\") pod \"43404f1c-c87c-4325-ab97-bffb6ea81e57\" (UID: \"43404f1c-c87c-4325-ab97-bffb6ea81e57\") "
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.019950    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43404f1c-c87c-4325-ab97-bffb6ea81e57-tls-certs\" (UniqueName: \"kubernetes.io/secret/43404f1c-c87c-4325-ab97-bffb6ea81e57-tls-certs\") pod \"43404f1c-c87c-4325-ab97-bffb6ea81e57\" (UID: \"43404f1c-c87c-4325-ab97-bffb6ea81e57\") "
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.019984    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-dshm\" (UniqueName: \"kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-dshm\") pod \"43404f1c-c87c-4325-ab97-bffb6ea81e57\" (UID: \"43404f1c-c87c-4325-ab97-bffb6ea81e57\") "
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.020067    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-tmp-dir\") pod \"4803807f-9312-40ac-badb-00c00d3f3bd3\" (UID: \"4803807f-9312-40ac-badb-00c00d3f3bd3\") "
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.020129    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/512b3ab1-45ba-4f96-9799-b129b25fca42-kube-api-access-25h56\" (UniqueName: \"kubernetes.io/projected/512b3ab1-45ba-4f96-9799-b129b25fca42-kube-api-access-25h56\") pod \"512b3ab1-45ba-4f96-9799-b129b25fca42\" (UID: \"512b3ab1-45ba-4f96-9799-b129b25fca42\") "
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.020166    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-model-cache\") pod \"512b3ab1-45ba-4f96-9799-b129b25fca42\" (UID: \"512b3ab1-45ba-4f96-9799-b129b25fca42\") "
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.020205    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43404f1c-c87c-4325-ab97-bffb6ea81e57-kube-api-access-7rwdf\" (UniqueName: \"kubernetes.io/projected/43404f1c-c87c-4325-ab97-bffb6ea81e57-kube-api-access-7rwdf\") pod \"43404f1c-c87c-4325-ab97-bffb6ea81e57\" (UID: \"43404f1c-c87c-4325-ab97-bffb6ea81e57\") "
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.020892    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-tmp-dir" pod "43404f1c-c87c-4325-ab97-bffb6ea81e57" (UID: "43404f1c-c87c-4325-ab97-bffb6ea81e57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.021202    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-home\" (UniqueName: \"kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-home\") pod \"a6c0d3aa-18c2-465b-846b-b82968acc780\" (UID: \"a6c0d3aa-18c2-465b-846b-b82968acc780\") "
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.021264    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4803807f-9312-40ac-badb-00c00d3f3bd3-kube-api-access-wvn7g\" (UniqueName: \"kubernetes.io/projected/4803807f-9312-40ac-badb-00c00d3f3bd3-kube-api-access-wvn7g\") pod \"4803807f-9312-40ac-badb-00c00d3f3bd3\" (UID: \"4803807f-9312-40ac-badb-00c00d3f3bd3\") "
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.021344    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-model-cache\") pod \"a6c0d3aa-18c2-465b-846b-b82968acc780\" (UID: \"a6c0d3aa-18c2-465b-846b-b82968acc780\") "
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.021371    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-dshm\") pod \"4803807f-9312-40ac-badb-00c00d3f3bd3\" (UID: \"4803807f-9312-40ac-badb-00c00d3f3bd3\") "
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.021402    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-tmp-dir\") pod \"a6c0d3aa-18c2-465b-846b-b82968acc780\" (UID: \"a6c0d3aa-18c2-465b-846b-b82968acc780\") "
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.021428    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4803807f-9312-40ac-badb-00c00d3f3bd3-tls-certs\" (UniqueName: \"kubernetes.io/secret/4803807f-9312-40ac-badb-00c00d3f3bd3-tls-certs\") pod \"4803807f-9312-40ac-badb-00c00d3f3bd3\" (UID: \"4803807f-9312-40ac-badb-00c00d3f3bd3\") "
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.021468    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-dshm\") pod \"a6c0d3aa-18c2-465b-846b-b82968acc780\" (UID: \"a6c0d3aa-18c2-465b-846b-b82968acc780\") "
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.021519    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-home\" (UniqueName: \"kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-home\") pod \"512b3ab1-45ba-4f96-9799-b129b25fca42\" (UID: \"512b3ab1-45ba-4f96-9799-b129b25fca42\") "
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.021552    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-model-cache\") pod \"43404f1c-c87c-4325-ab97-bffb6ea81e57\" (UID: \"43404f1c-c87c-4325-ab97-bffb6ea81e57\") "
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.021610    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-tmp-dir\") pod \"512b3ab1-45ba-4f96-9799-b129b25fca42\" (UID: \"512b3ab1-45ba-4f96-9799-b129b25fca42\") "
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.021642    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-home\" (UniqueName: \"kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-home\") pod \"43404f1c-c87c-4325-ab97-bffb6ea81e57\" (UID: \"43404f1c-c87c-4325-ab97-bffb6ea81e57\") "
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.021671    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-dshm\" (UniqueName: \"kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-dshm\") pod \"512b3ab1-45ba-4f96-9799-b129b25fca42\" (UID: \"512b3ab1-45ba-4f96-9799-b129b25fca42\") "
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.021702    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6c0d3aa-18c2-465b-846b-b82968acc780-kube-api-access-lgdf5\" (UniqueName: \"kubernetes.io/projected/a6c0d3aa-18c2-465b-846b-b82968acc780-kube-api-access-lgdf5\") pod \"a6c0d3aa-18c2-465b-846b-b82968acc780\" (UID: \"a6c0d3aa-18c2-465b-846b-b82968acc780\") "
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.021728    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/512b3ab1-45ba-4f96-9799-b129b25fca42-tls-certs\" (UniqueName: \"kubernetes.io/secret/512b3ab1-45ba-4f96-9799-b129b25fca42-tls-certs\") pod \"512b3ab1-45ba-4f96-9799-b129b25fca42\" (UID: \"512b3ab1-45ba-4f96-9799-b129b25fca42\") "
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.021752    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-model-cache\") pod \"4803807f-9312-40ac-badb-00c00d3f3bd3\" (UID: \"4803807f-9312-40ac-badb-00c00d3f3bd3\") "
Nov 03 22:22:31.022312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.021781    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-home\" (UniqueName: \"kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-home\") pod \"4803807f-9312-40ac-badb-00c00d3f3bd3\" (UID: \"4803807f-9312-40ac-badb-00c00d3f3bd3\") "
Nov 03 22:22:31.023762 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.022788    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-home" pod "512b3ab1-45ba-4f96-9799-b129b25fca42" (UID: "512b3ab1-45ba-4f96-9799-b129b25fca42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:31.023762 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.022807    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-home" pod "a6c0d3aa-18c2-465b-846b-b82968acc780" (UID: "a6c0d3aa-18c2-465b-846b-b82968acc780"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:31.023762 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.023003    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-tmp-dir" pod "4803807f-9312-40ac-badb-00c00d3f3bd3" (UID: "4803807f-9312-40ac-badb-00c00d3f3bd3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:31.023762 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.023111    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-home" pod "43404f1c-c87c-4325-ab97-bffb6ea81e57" (UID: "43404f1c-c87c-4325-ab97-bffb6ea81e57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:31.023762 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.023310    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-model-cache" pod "a6c0d3aa-18c2-465b-846b-b82968acc780" (UID: "a6c0d3aa-18c2-465b-846b-b82968acc780"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:31.023762 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.023571    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-model-cache" pod "43404f1c-c87c-4325-ab97-bffb6ea81e57" (UID: "43404f1c-c87c-4325-ab97-bffb6ea81e57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:31.023762 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.023749    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-tmp-dir" pod "512b3ab1-45ba-4f96-9799-b129b25fca42" (UID: "512b3ab1-45ba-4f96-9799-b129b25fca42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:31.029486 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.029042    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-tmp-dir" pod "a6c0d3aa-18c2-465b-846b-b82968acc780" (UID: "a6c0d3aa-18c2-465b-846b-b82968acc780"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:31.029486 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.029223    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-model-cache" pod "512b3ab1-45ba-4f96-9799-b129b25fca42" (UID: "512b3ab1-45ba-4f96-9799-b129b25fca42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:31.035433 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.035040    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43404f1c-c87c-4325-ab97-bffb6ea81e57-tls-certs" pod "43404f1c-c87c-4325-ab97-bffb6ea81e57" (UID: "43404f1c-c87c-4325-ab97-bffb6ea81e57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:22:31.035433 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.035288    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-home" pod "4803807f-9312-40ac-badb-00c00d3f3bd3" (UID: "4803807f-9312-40ac-badb-00c00d3f3bd3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:31.035433 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.035380    5053 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-tmp-dir\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:31.035710 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.035483    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-model-cache" pod "4803807f-9312-40ac-badb-00c00d3f3bd3" (UID: "4803807f-9312-40ac-badb-00c00d3f3bd3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:31.036250 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.036156    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4803807f-9312-40ac-badb-00c00d3f3bd3-kube-api-access-wvn7g" pod "4803807f-9312-40ac-badb-00c00d3f3bd3" (UID: "4803807f-9312-40ac-badb-00c00d3f3bd3"). InnerVolumeSpecName "kube-api-access-wvn7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:31.036542 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.036500    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4803807f-9312-40ac-badb-00c00d3f3bd3-tls-certs" pod "4803807f-9312-40ac-badb-00c00d3f3bd3" (UID: "4803807f-9312-40ac-badb-00c00d3f3bd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:22:31.036968 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.036942    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c0d3aa-18c2-465b-846b-b82968acc780-kube-api-access-lgdf5" pod "a6c0d3aa-18c2-465b-846b-b82968acc780" (UID: "a6c0d3aa-18c2-465b-846b-b82968acc780"). InnerVolumeSpecName "kube-api-access-lgdf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:31.037150 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.037116    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-dshm" pod "4803807f-9312-40ac-badb-00c00d3f3bd3" (UID: "4803807f-9312-40ac-badb-00c00d3f3bd3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:31.038146 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.038116    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43404f1c-c87c-4325-ab97-bffb6ea81e57-kube-api-access-7rwdf" pod "43404f1c-c87c-4325-ab97-bffb6ea81e57" (UID: "43404f1c-c87c-4325-ab97-bffb6ea81e57"). InnerVolumeSpecName "kube-api-access-7rwdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:31.038641 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.038617    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/512b3ab1-45ba-4f96-9799-b129b25fca42-tls-certs" pod "512b3ab1-45ba-4f96-9799-b129b25fca42" (UID: "512b3ab1-45ba-4f96-9799-b129b25fca42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:22:31.044685 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.040731    5053 scope.go:122] "RemoveContainer" containerID="05761c88f26f13861c837676cdcec7871393c6846daddbe2d7fa5993e9b8a78e"
Nov 03 22:22:31.044685 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:22:31.041173    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05761c88f26f13861c837676cdcec7871393c6846daddbe2d7fa5993e9b8a78e\": container with ID starting with 05761c88f26f13861c837676cdcec7871393c6846daddbe2d7fa5993e9b8a78e not found: ID does not exist" containerID="05761c88f26f13861c837676cdcec7871393c6846daddbe2d7fa5993e9b8a78e"
Nov 03 22:22:31.044685 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.041199    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05761c88f26f13861c837676cdcec7871393c6846daddbe2d7fa5993e9b8a78e"} err="failed to get container status \"05761c88f26f13861c837676cdcec7871393c6846daddbe2d7fa5993e9b8a78e\": rpc error: code = NotFound desc = could not find container \"05761c88f26f13861c837676cdcec7871393c6846daddbe2d7fa5993e9b8a78e\": container with ID starting with 05761c88f26f13861c837676cdcec7871393c6846daddbe2d7fa5993e9b8a78e not found: ID does not exist"
Nov 03 22:22:31.044685 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.044651    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-dshm" pod "43404f1c-c87c-4325-ab97-bffb6ea81e57" (UID: "43404f1c-c87c-4325-ab97-bffb6ea81e57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:31.044685 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.044658    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-dshm" pod "a6c0d3aa-18c2-465b-846b-b82968acc780" (UID: "a6c0d3aa-18c2-465b-846b-b82968acc780"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:31.045381 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.045361    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6c0d3aa-18c2-465b-846b-b82968acc780-tls-certs" pod "a6c0d3aa-18c2-465b-846b-b82968acc780" (UID: "a6c0d3aa-18c2-465b-846b-b82968acc780"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:22:31.045482 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.045379    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-dshm" pod "512b3ab1-45ba-4f96-9799-b129b25fca42" (UID: "512b3ab1-45ba-4f96-9799-b129b25fca42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:31.046216 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.046193    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/512b3ab1-45ba-4f96-9799-b129b25fca42-kube-api-access-25h56" pod "512b3ab1-45ba-4f96-9799-b129b25fca42" (UID: "512b3ab1-45ba-4f96-9799-b129b25fca42"). InnerVolumeSpecName "kube-api-access-25h56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:31.138799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.137046    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rwdf\" (UniqueName: \"kubernetes.io/projected/43404f1c-c87c-4325-ab97-bffb6ea81e57-kube-api-access-7rwdf\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:31.138799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.137089    5053 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-home\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:31.138799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.137103    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvn7g\" (UniqueName: \"kubernetes.io/projected/4803807f-9312-40ac-badb-00c00d3f3bd3-kube-api-access-wvn7g\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:31.138799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.137118    5053 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-model-cache\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:31.138799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.137134    5053 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-dshm\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:31.138799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.137148    5053 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-tmp-dir\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:31.138799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.137160    5053 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4803807f-9312-40ac-badb-00c00d3f3bd3-tls-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:31.138799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.137172    5053 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c0d3aa-18c2-465b-846b-b82968acc780-dshm\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:31.138799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.137183    5053 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-home\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:31.138799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.137195    5053 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-model-cache\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:31.138799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.137209    5053 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-tmp-dir\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:31.138799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.137220    5053 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-home\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:31.138799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.137248    5053 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-dshm\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:31.138799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.137260    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgdf5\" (UniqueName: \"kubernetes.io/projected/a6c0d3aa-18c2-465b-846b-b82968acc780-kube-api-access-lgdf5\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:31.138799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.137275    5053 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/512b3ab1-45ba-4f96-9799-b129b25fca42-tls-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:31.138799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.137288    5053 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-model-cache\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:31.138799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.137300    5053 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-home\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:31.138799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.137321    5053 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c0d3aa-18c2-465b-846b-b82968acc780-tls-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:31.138799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.137334    5053 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43404f1c-c87c-4325-ab97-bffb6ea81e57-tls-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:31.138799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.137346    5053 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43404f1c-c87c-4325-ab97-bffb6ea81e57-dshm\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:31.138799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.137356    5053 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4803807f-9312-40ac-badb-00c00d3f3bd3-tmp-dir\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:31.138799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.137366    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25h56\" (UniqueName: \"kubernetes.io/projected/512b3ab1-45ba-4f96-9799-b129b25fca42-kube-api-access-25h56\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:31.138799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.137378    5053 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/512b3ab1-45ba-4f96-9799-b129b25fca42-model-cache\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:31.523131 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.523093    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"]
Nov 03 22:22:31.525510 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.525485    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6de8e6/premium-w4-6de8e6-kserve-76bc9fd78-lbkxx"]
Nov 03 22:22:31.639606 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.639570    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"]
Nov 03 22:22:31.643116 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.643090    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6de8e6/simulator-w4-6de8e6-kserve-5fcf67d9f5-k272n"]
Nov 03 22:22:31.666452 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.666401    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="512b3ab1-45ba-4f96-9799-b129b25fca42" path="/var/lib/kubelet/pods/512b3ab1-45ba-4f96-9799-b129b25fca42/volumes"
Nov 03 22:22:31.666948 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.666927    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6c0d3aa-18c2-465b-846b-b82968acc780" path="/var/lib/kubelet/pods/a6c0d3aa-18c2-465b-846b-b82968acc780/volumes"
Nov 03 22:22:31.757294 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.757250    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"]
Nov 03 22:22:31.757505 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.757322    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8"]
Nov 03 22:22:31.828254 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.828125    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6de8e6/distinct-w4-6de8e6-kserve-65d58c4d45-j8tx8" podUID="43404f1c-c87c-4325-ab97-bffb6ea81e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:22:31.867155 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.867111    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"]
Nov 03 22:22:31.871332 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.871302    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6de8e6/distinct-2-w4-6de8e6-kserve-6978d99d58-wrbd6"]
Nov 03 22:22:31.874743 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.874721    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d6fe726-4j2sv"]
Nov 03 22:22:31.875250 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.875224    5053 cpu_manager.go:414] "containerMap: removing container" podUID="a6c0d3aa-18c2-465b-846b-b82968acc780" containerName="main"
Nov 03 22:22:31.875298 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.875254    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6c0d3aa-18c2-465b-846b-b82968acc780" containerName="main"
Nov 03 22:22:31.875298 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.875278    5053 cpu_manager.go:414] "containerMap: removing container" podUID="4803807f-9312-40ac-badb-00c00d3f3bd3" containerName="main"
Nov 03 22:22:31.875298 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.875283    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4803807f-9312-40ac-badb-00c00d3f3bd3" containerName="main"
Nov 03 22:22:31.875397 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.875302    5053 cpu_manager.go:414] "containerMap: removing container" podUID="43404f1c-c87c-4325-ab97-bffb6ea81e57" containerName="main"
Nov 03 22:22:31.875397 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.875307    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43404f1c-c87c-4325-ab97-bffb6ea81e57" containerName="main"
Nov 03 22:22:31.875397 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.875315    5053 cpu_manager.go:414] "containerMap: removing container" podUID="f4ad58fc-198d-4b28-9f7a-380f56bc6257" containerName="revoke-keys"
Nov 03 22:22:31.875397 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.875320    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ad58fc-198d-4b28-9f7a-380f56bc6257" containerName="revoke-keys"
Nov 03 22:22:31.875397 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.875329    5053 cpu_manager.go:414] "containerMap: removing container" podUID="512b3ab1-45ba-4f96-9799-b129b25fca42" containerName="main"
Nov 03 22:22:31.875397 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.875333    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="512b3ab1-45ba-4f96-9799-b129b25fca42" containerName="main"
Nov 03 22:22:31.875583 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.875406    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6c0d3aa-18c2-465b-846b-b82968acc780" containerName="main"
Nov 03 22:22:31.875583 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.875416    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="4803807f-9312-40ac-badb-00c00d3f3bd3" containerName="main"
Nov 03 22:22:31.875583 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.875425    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="43404f1c-c87c-4325-ab97-bffb6ea81e57" containerName="main"
Nov 03 22:22:31.875583 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.875431    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ad58fc-198d-4b28-9f7a-380f56bc6257" containerName="revoke-keys"
Nov 03 22:22:31.875583 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.875437    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="512b3ab1-45ba-4f96-9799-b129b25fca42" containerName="main"
Nov 03 22:22:31.880500 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.880473    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6fe726-4j2sv"
Nov 03 22:22:31.937759 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.937727    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d6fe726-4j2sv"]
Nov 03 22:22:31.940977 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.940945    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:22:31.941162 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.941144    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:22:31.961115 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.961080    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90d53cf2-e5d8-4561-b87a-bcbd9485f141-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d6fe726-4j2sv\" (UID: \"90d53cf2-e5d8-4561-b87a-bcbd9485f141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6fe726-4j2sv"
Nov 03 22:22:31.961327 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.961131    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztqqm\" (UniqueName: \"kubernetes.io/projected/90d53cf2-e5d8-4561-b87a-bcbd9485f141-kube-api-access-ztqqm\") pod \"maas-api-revoke-sub-6d6fe726-4j2sv\" (UID: \"90d53cf2-e5d8-4561-b87a-bcbd9485f141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6fe726-4j2sv"
Nov 03 22:22:31.961327 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:31.961284    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90d53cf2-e5d8-4561-b87a-bcbd9485f141-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d6fe726-4j2sv\" (UID: \"90d53cf2-e5d8-4561-b87a-bcbd9485f141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6fe726-4j2sv"
Nov 03 22:22:32.062329 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:32.062290    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90d53cf2-e5d8-4561-b87a-bcbd9485f141-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d6fe726-4j2sv\" (UID: \"90d53cf2-e5d8-4561-b87a-bcbd9485f141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6fe726-4j2sv"
Nov 03 22:22:32.062517 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:32.062488    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90d53cf2-e5d8-4561-b87a-bcbd9485f141-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d6fe726-4j2sv\" (UID: \"90d53cf2-e5d8-4561-b87a-bcbd9485f141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6fe726-4j2sv"
Nov 03 22:22:32.062576 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:32.062519    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztqqm\" (UniqueName: \"kubernetes.io/projected/90d53cf2-e5d8-4561-b87a-bcbd9485f141-kube-api-access-ztqqm\") pod \"maas-api-revoke-sub-6d6fe726-4j2sv\" (UID: \"90d53cf2-e5d8-4561-b87a-bcbd9485f141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6fe726-4j2sv"
Nov 03 22:22:32.063066 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:32.063044    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90d53cf2-e5d8-4561-b87a-bcbd9485f141-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d6fe726-4j2sv\" (UID: \"90d53cf2-e5d8-4561-b87a-bcbd9485f141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6fe726-4j2sv"
Nov 03 22:22:32.065098 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:32.065070    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90d53cf2-e5d8-4561-b87a-bcbd9485f141-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d6fe726-4j2sv\" (UID: \"90d53cf2-e5d8-4561-b87a-bcbd9485f141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6fe726-4j2sv"
Nov 03 22:22:32.120922 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:32.120853    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztqqm\" (UniqueName: \"kubernetes.io/projected/90d53cf2-e5d8-4561-b87a-bcbd9485f141-kube-api-access-ztqqm\") pod \"maas-api-revoke-sub-6d6fe726-4j2sv\" (UID: \"90d53cf2-e5d8-4561-b87a-bcbd9485f141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6fe726-4j2sv"
Nov 03 22:22:32.194069 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:32.194031    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6fe726-4j2sv"
Nov 03 22:22:32.570702 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:32.570677    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d6fe726-4j2sv"]
Nov 03 22:22:32.572509 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:22:32.572481    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d53cf2_e5d8_4561_b87a_bcbd9485f141.slice/crio-4290404e28a74292406a21246eb023740b5294c0bbf60dd3e51b9497733f7602 WatchSource:0}: Error finding container 4290404e28a74292406a21246eb023740b5294c0bbf60dd3e51b9497733f7602: Status 404 returned error can't find the container with id 4290404e28a74292406a21246eb023740b5294c0bbf60dd3e51b9497733f7602
Nov 03 22:22:32.990160 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:32.990128    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6fe726-4j2sv" event={"ID":"90d53cf2-e5d8-4561-b87a-bcbd9485f141","Type":"ContainerStarted","Data":"5f5ce8c23af077372df58ec59c0c6e3e2e23a4d3f95b1f32b27f73c368c99b17"}
Nov 03 22:22:32.990565 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:32.990175    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6fe726-4j2sv" event={"ID":"90d53cf2-e5d8-4561-b87a-bcbd9485f141","Type":"ContainerStarted","Data":"4290404e28a74292406a21246eb023740b5294c0bbf60dd3e51b9497733f7602"}
Nov 03 22:22:33.104218 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:33.104159    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6fe726-4j2sv" podStartSLOduration=2.104143513 podStartE2EDuration="2.104143513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:33.10305641 +0000 UTC m=+4067.596978283" watchObservedRunningTime="2026-11-03 22:22:33.104143513 +0000 UTC m=+4067.598065418"
Nov 03 22:22:33.666860 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:33.666827    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43404f1c-c87c-4325-ab97-bffb6ea81e57" path="/var/lib/kubelet/pods/43404f1c-c87c-4325-ab97-bffb6ea81e57/volumes"
Nov 03 22:22:33.667321 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:33.667305    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4803807f-9312-40ac-badb-00c00d3f3bd3" path="/var/lib/kubelet/pods/4803807f-9312-40ac-badb-00c00d3f3bd3/volumes"
Nov 03 22:22:33.763428 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:33.763339    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b" podUID="b4036e26-b7be-4c14-bd68-4863da3c3fe0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:22:33.763428 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:33.763387    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9" podUID="4cee4995-fb03-43bc-a63d-f764dba58a39" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:22:33.999138 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:33.999105    5053 generic.go:356] "Generic (PLEG): container finished" podID="90d53cf2-e5d8-4561-b87a-bcbd9485f141" containerID="5f5ce8c23af077372df58ec59c0c6e3e2e23a4d3f95b1f32b27f73c368c99b17" exitCode=0
Nov 03 22:22:33.999554 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:33.999180    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6fe726-4j2sv" event={"ID":"90d53cf2-e5d8-4561-b87a-bcbd9485f141","Type":"ContainerDied","Data":"5f5ce8c23af077372df58ec59c0c6e3e2e23a4d3f95b1f32b27f73c368c99b17"}
Nov 03 22:22:35.178688 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:35.178658    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6fe726-4j2sv"
Nov 03 22:22:35.300708 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:35.300623    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90d53cf2-e5d8-4561-b87a-bcbd9485f141-kube-api-access-ztqqm\" (UniqueName: \"kubernetes.io/projected/90d53cf2-e5d8-4561-b87a-bcbd9485f141-kube-api-access-ztqqm\") pod \"90d53cf2-e5d8-4561-b87a-bcbd9485f141\" (UID: \"90d53cf2-e5d8-4561-b87a-bcbd9485f141\") "
Nov 03 22:22:35.300907 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:35.300768    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90d53cf2-e5d8-4561-b87a-bcbd9485f141-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90d53cf2-e5d8-4561-b87a-bcbd9485f141-maas-api-openshift-ca\") pod \"90d53cf2-e5d8-4561-b87a-bcbd9485f141\" (UID: \"90d53cf2-e5d8-4561-b87a-bcbd9485f141\") "
Nov 03 22:22:35.300907 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:35.300863    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90d53cf2-e5d8-4561-b87a-bcbd9485f141-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90d53cf2-e5d8-4561-b87a-bcbd9485f141-maas-api-xks-ca\") pod \"90d53cf2-e5d8-4561-b87a-bcbd9485f141\" (UID: \"90d53cf2-e5d8-4561-b87a-bcbd9485f141\") "
Nov 03 22:22:35.301528 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:35.301490    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90d53cf2-e5d8-4561-b87a-bcbd9485f141-maas-api-openshift-ca" pod "90d53cf2-e5d8-4561-b87a-bcbd9485f141" (UID: "90d53cf2-e5d8-4561-b87a-bcbd9485f141"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:22:35.303950 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:35.303922    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d53cf2-e5d8-4561-b87a-bcbd9485f141-kube-api-access-ztqqm" pod "90d53cf2-e5d8-4561-b87a-bcbd9485f141" (UID: "90d53cf2-e5d8-4561-b87a-bcbd9485f141"). InnerVolumeSpecName "kube-api-access-ztqqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:35.304366 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:35.304336    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90d53cf2-e5d8-4561-b87a-bcbd9485f141-maas-api-xks-ca" pod "90d53cf2-e5d8-4561-b87a-bcbd9485f141" (UID: "90d53cf2-e5d8-4561-b87a-bcbd9485f141"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:22:35.402183 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:35.402147    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztqqm\" (UniqueName: \"kubernetes.io/projected/90d53cf2-e5d8-4561-b87a-bcbd9485f141-kube-api-access-ztqqm\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:35.402183 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:35.402178    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90d53cf2-e5d8-4561-b87a-bcbd9485f141-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:35.402183 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:35.402189    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90d53cf2-e5d8-4561-b87a-bcbd9485f141-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:36.013503 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:36.013468    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6fe726-4j2sv"
Nov 03 22:22:36.013717 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:36.013493    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6fe726-4j2sv" event={"ID":"90d53cf2-e5d8-4561-b87a-bcbd9485f141","Type":"ContainerDied","Data":"4290404e28a74292406a21246eb023740b5294c0bbf60dd3e51b9497733f7602"}
Nov 03 22:22:36.013717 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:36.013557    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4290404e28a74292406a21246eb023740b5294c0bbf60dd3e51b9497733f7602"
Nov 03 22:22:38.280100 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:38.280063    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"]
Nov 03 22:22:38.280508 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:38.280302    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z" podUID="f9207d7a-c808-4e41-8e12-c3219c4f8215" containerName="istio-proxy" containerID="cri-o://17504f2718462eca7940beed493d3a0aaf7bb6c65be1abb1bfb081894d51fa00" gracePeriod=30
Nov 03 22:22:39.721770 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:39.721736    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l"]
Nov 03 22:22:39.771671 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:39.771639    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e15bbc-l955l"]
Nov 03 22:22:39.901931 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:39.901895    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"]
Nov 03 22:22:39.902168 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:39.902123    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs" podUID="d17d1dfe-8556-43d4-b67f-d0e628552f0b" containerName="istio-proxy" containerID="cri-o://cf529980f5c953275d110a322ade4ef906f85992c34846c724e61fa8ba713068" gracePeriod=30
Nov 03 22:22:40.928343 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:22:40.928304    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:22:41.018725 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:41.018684    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp"]
Nov 03 22:22:41.019319 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:41.019303    5053 cpu_manager.go:414] "containerMap: removing container" podUID="90d53cf2-e5d8-4561-b87a-bcbd9485f141" containerName="revoke-keys"
Nov 03 22:22:41.019362 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:41.019320    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90d53cf2-e5d8-4561-b87a-bcbd9485f141" containerName="revoke-keys"
Nov 03 22:22:41.019414 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:41.019404    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="90d53cf2-e5d8-4561-b87a-bcbd9485f141" containerName="revoke-keys"
Nov 03 22:22:41.023831 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:41.023815    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp"
Nov 03 22:22:41.069874 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:41.069836    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rsdh\" (UniqueName: \"kubernetes.io/projected/bb874fe1-5610-449f-8c84-25cbc329667a-kube-api-access-8rsdh\") pod \"maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp\" (UID: \"bb874fe1-5610-449f-8c84-25cbc329667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp"
Nov 03 22:22:41.070050 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:41.069949    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb874fe1-5610-449f-8c84-25cbc329667a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp\" (UID: \"bb874fe1-5610-449f-8c84-25cbc329667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp"
Nov 03 22:22:41.070050 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:41.069986    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb874fe1-5610-449f-8c84-25cbc329667a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp\" (UID: \"bb874fe1-5610-449f-8c84-25cbc329667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp"
Nov 03 22:22:41.076279 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:41.076221    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:22:41.079187 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:41.079156    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp"]
Nov 03 22:22:41.125415 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:41.125372    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:22:41.170788 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:41.170750    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rsdh\" (UniqueName: \"kubernetes.io/projected/bb874fe1-5610-449f-8c84-25cbc329667a-kube-api-access-8rsdh\") pod \"maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp\" (UID: \"bb874fe1-5610-449f-8c84-25cbc329667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp"
Nov 03 22:22:41.171004 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:41.170855    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb874fe1-5610-449f-8c84-25cbc329667a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp\" (UID: \"bb874fe1-5610-449f-8c84-25cbc329667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp"
Nov 03 22:22:41.171004 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:41.170887    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb874fe1-5610-449f-8c84-25cbc329667a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp\" (UID: \"bb874fe1-5610-449f-8c84-25cbc329667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp"
Nov 03 22:22:41.171857 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:41.171832    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb874fe1-5610-449f-8c84-25cbc329667a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp\" (UID: \"bb874fe1-5610-449f-8c84-25cbc329667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp"
Nov 03 22:22:41.173493 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:41.173469    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb874fe1-5610-449f-8c84-25cbc329667a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp\" (UID: \"bb874fe1-5610-449f-8c84-25cbc329667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp"
Nov 03 22:22:41.227232 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:41.227200    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rsdh\" (UniqueName: \"kubernetes.io/projected/bb874fe1-5610-449f-8c84-25cbc329667a-kube-api-access-8rsdh\") pod \"maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp\" (UID: \"bb874fe1-5610-449f-8c84-25cbc329667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp"
Nov 03 22:22:41.337608 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:41.337577    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp"
Nov 03 22:22:41.666660 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:41.666580    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab33391e-7f3d-4b28-a61b-a311284df2ab" path="/var/lib/kubelet/pods/ab33391e-7f3d-4b28-a61b-a311284df2ab/volumes"
Nov 03 22:22:41.711211 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:41.711179    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp"]
Nov 03 22:22:41.712975 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:22:41.712950    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb874fe1_5610_449f_8c84_25cbc329667a.slice/crio-c46b5d93d5164f624f7ecc5caec4f84953d7e8785dc40cc68d9ef08c998c853f WatchSource:0}: Error finding container c46b5d93d5164f624f7ecc5caec4f84953d7e8785dc40cc68d9ef08c998c853f: Status 404 returned error can't find the container with id c46b5d93d5164f624f7ecc5caec4f84953d7e8785dc40cc68d9ef08c998c853f
Nov 03 22:22:42.056698 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:42.056660    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp" event={"ID":"bb874fe1-5610-449f-8c84-25cbc329667a","Type":"ContainerStarted","Data":"6b26c62e9c73398bf8ad66485f94c79c3a91dfdb20c86127ff378339b5519522"}
Nov 03 22:22:42.057106 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:42.056720    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp" event={"ID":"bb874fe1-5610-449f-8c84-25cbc329667a","Type":"ContainerStarted","Data":"c46b5d93d5164f624f7ecc5caec4f84953d7e8785dc40cc68d9ef08c998c853f"}
Nov 03 22:22:42.169961 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:42.169856    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp" podStartSLOduration=2.169840112 podStartE2EDuration="2.169840112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:42.167985254 +0000 UTC m=+4076.661907133" watchObservedRunningTime="2026-11-03 22:22:42.169840112 +0000 UTC m=+4076.663761983"
Nov 03 22:22:43.065661 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.065628    5053 generic.go:356] "Generic (PLEG): container finished" podID="bb874fe1-5610-449f-8c84-25cbc329667a" containerID="6b26c62e9c73398bf8ad66485f94c79c3a91dfdb20c86127ff378339b5519522" exitCode=0
Nov 03 22:22:43.066106 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.065704    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp" event={"ID":"bb874fe1-5610-449f-8c84-25cbc329667a","Type":"ContainerDied","Data":"6b26c62e9c73398bf8ad66485f94c79c3a91dfdb20c86127ff378339b5519522"}
Nov 03 22:22:43.538677 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.538650    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 22:22:43.596677 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.596643    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-token\" (UniqueName: \"kubernetes.io/projected/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-token\") pod \"f9207d7a-c808-4e41-8e12-c3219c4f8215\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") "
Nov 03 22:22:43.596889 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.596689    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-credential-socket\") pod \"f9207d7a-c808-4e41-8e12-c3219c4f8215\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") "
Nov 03 22:22:43.596889 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.596722    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-data\") pod \"f9207d7a-c808-4e41-8e12-c3219c4f8215\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") "
Nov 03 22:22:43.596889 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.596759    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9207d7a-c808-4e41-8e12-c3219c4f8215-kube-api-access-btnkn\" (UniqueName: \"kubernetes.io/projected/f9207d7a-c808-4e41-8e12-c3219c4f8215-kube-api-access-btnkn\") pod \"f9207d7a-c808-4e41-8e12-c3219c4f8215\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") "
Nov 03 22:22:43.596889 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.596866    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-podinfo\") pod \"f9207d7a-c808-4e41-8e12-c3219c4f8215\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") "
Nov 03 22:22:43.597101 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.596893    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9207d7a-c808-4e41-8e12-c3219c4f8215-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9207d7a-c808-4e41-8e12-c3219c4f8215-istiod-ca-cert\") pod \"f9207d7a-c808-4e41-8e12-c3219c4f8215\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") "
Nov 03 22:22:43.597101 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.596941    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-workload-certs\") pod \"f9207d7a-c808-4e41-8e12-c3219c4f8215\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") "
Nov 03 22:22:43.597101 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.596995    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-envoy\") pod \"f9207d7a-c808-4e41-8e12-c3219c4f8215\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") "
Nov 03 22:22:43.597101 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.597044    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-workload-socket\") pod \"f9207d7a-c808-4e41-8e12-c3219c4f8215\" (UID: \"f9207d7a-c808-4e41-8e12-c3219c4f8215\") "
Nov 03 22:22:43.597101 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.597074    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-credential-socket" pod "f9207d7a-c808-4e41-8e12-c3219c4f8215" (UID: "f9207d7a-c808-4e41-8e12-c3219c4f8215"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:43.597101 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.597089    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-data" pod "f9207d7a-c808-4e41-8e12-c3219c4f8215" (UID: "f9207d7a-c808-4e41-8e12-c3219c4f8215"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:43.597446 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.597339    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-workload-certs" pod "f9207d7a-c808-4e41-8e12-c3219c4f8215" (UID: "f9207d7a-c808-4e41-8e12-c3219c4f8215"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:43.597518 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.597490    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-workload-socket" pod "f9207d7a-c808-4e41-8e12-c3219c4f8215" (UID: "f9207d7a-c808-4e41-8e12-c3219c4f8215"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:43.597518 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.597428    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9207d7a-c808-4e41-8e12-c3219c4f8215-istiod-ca-cert" pod "f9207d7a-c808-4e41-8e12-c3219c4f8215" (UID: "f9207d7a-c808-4e41-8e12-c3219c4f8215"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:22:43.597620 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.597560    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:43.597620 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.597580    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:43.597620 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.597597    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:43.599814 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.599761    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-podinfo" pod "f9207d7a-c808-4e41-8e12-c3219c4f8215" (UID: "f9207d7a-c808-4e41-8e12-c3219c4f8215"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:22:43.599935 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.599836    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-envoy" pod "f9207d7a-c808-4e41-8e12-c3219c4f8215" (UID: "f9207d7a-c808-4e41-8e12-c3219c4f8215"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:43.599935 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.599894    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-token" pod "f9207d7a-c808-4e41-8e12-c3219c4f8215" (UID: "f9207d7a-c808-4e41-8e12-c3219c4f8215"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:43.599935 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.599926    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9207d7a-c808-4e41-8e12-c3219c4f8215-kube-api-access-btnkn" pod "f9207d7a-c808-4e41-8e12-c3219c4f8215" (UID: "f9207d7a-c808-4e41-8e12-c3219c4f8215"). InnerVolumeSpecName "kube-api-access-btnkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:43.698574 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.698544    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:43.698574 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.698571    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9207d7a-c808-4e41-8e12-c3219c4f8215-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:43.698788 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.698584    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:43.698788 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.698594    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9207d7a-c808-4e41-8e12-c3219c4f8215-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:43.698788 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.698602    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9207d7a-c808-4e41-8e12-c3219c4f8215-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:43.698788 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.698611    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btnkn\" (UniqueName: \"kubernetes.io/projected/f9207d7a-c808-4e41-8e12-c3219c4f8215-kube-api-access-btnkn\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:43.763833 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.763799    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9" podUID="4cee4995-fb03-43bc-a63d-f764dba58a39" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:22:43.764060 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:43.763799    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b" podUID="b4036e26-b7be-4c14-bd68-4863da3c3fe0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:22:44.075165 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:44.075133    5053 generic.go:356] "Generic (PLEG): container finished" podID="f9207d7a-c808-4e41-8e12-c3219c4f8215" containerID="17504f2718462eca7940beed493d3a0aaf7bb6c65be1abb1bfb081894d51fa00" exitCode=0
Nov 03 22:22:44.075638 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:44.075190    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"
Nov 03 22:22:44.075638 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:44.075222    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z" event={"ID":"f9207d7a-c808-4e41-8e12-c3219c4f8215","Type":"ContainerDied","Data":"17504f2718462eca7940beed493d3a0aaf7bb6c65be1abb1bfb081894d51fa00"}
Nov 03 22:22:44.075638 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:44.075303    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z" event={"ID":"f9207d7a-c808-4e41-8e12-c3219c4f8215","Type":"ContainerDied","Data":"2a9b16e464138fc47c757a7618d3e24a98c56f8a6d0e7d9a78d2a43776761b82"}
Nov 03 22:22:44.075638 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:44.075331    5053 scope.go:122] "RemoveContainer" containerID="17504f2718462eca7940beed493d3a0aaf7bb6c65be1abb1bfb081894d51fa00"
Nov 03 22:22:44.087127 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:44.087105    5053 scope.go:122] "RemoveContainer" containerID="411eab61a7165a43acc3fb20510a57505c42e0aea66d11c93c5106ba4cb37d1f"
Nov 03 22:22:44.097033 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:44.097013    5053 scope.go:122] "RemoveContainer" containerID="17504f2718462eca7940beed493d3a0aaf7bb6c65be1abb1bfb081894d51fa00"
Nov 03 22:22:44.097398 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:22:44.097377    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17504f2718462eca7940beed493d3a0aaf7bb6c65be1abb1bfb081894d51fa00\": container with ID starting with 17504f2718462eca7940beed493d3a0aaf7bb6c65be1abb1bfb081894d51fa00 not found: ID does not exist" containerID="17504f2718462eca7940beed493d3a0aaf7bb6c65be1abb1bfb081894d51fa00"
Nov 03 22:22:44.097474 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:44.097406    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17504f2718462eca7940beed493d3a0aaf7bb6c65be1abb1bfb081894d51fa00"} err="failed to get container status \"17504f2718462eca7940beed493d3a0aaf7bb6c65be1abb1bfb081894d51fa00\": rpc error: code = NotFound desc = could not find container \"17504f2718462eca7940beed493d3a0aaf7bb6c65be1abb1bfb081894d51fa00\": container with ID starting with 17504f2718462eca7940beed493d3a0aaf7bb6c65be1abb1bfb081894d51fa00 not found: ID does not exist"
Nov 03 22:22:44.097474 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:44.097427    5053 scope.go:122] "RemoveContainer" containerID="411eab61a7165a43acc3fb20510a57505c42e0aea66d11c93c5106ba4cb37d1f"
Nov 03 22:22:44.097734 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:22:44.097715    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"411eab61a7165a43acc3fb20510a57505c42e0aea66d11c93c5106ba4cb37d1f\": container with ID starting with 411eab61a7165a43acc3fb20510a57505c42e0aea66d11c93c5106ba4cb37d1f not found: ID does not exist" containerID="411eab61a7165a43acc3fb20510a57505c42e0aea66d11c93c5106ba4cb37d1f"
Nov 03 22:22:44.097796 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:44.097744    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"411eab61a7165a43acc3fb20510a57505c42e0aea66d11c93c5106ba4cb37d1f"} err="failed to get container status \"411eab61a7165a43acc3fb20510a57505c42e0aea66d11c93c5106ba4cb37d1f\": rpc error: code = NotFound desc = could not find container \"411eab61a7165a43acc3fb20510a57505c42e0aea66d11c93c5106ba4cb37d1f\": container with ID starting with 411eab61a7165a43acc3fb20510a57505c42e0aea66d11c93c5106ba4cb37d1f not found: ID does not exist"
Nov 03 22:22:44.200512 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:44.200485    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp"
Nov 03 22:22:44.304260 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:44.304206    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb874fe1-5610-449f-8c84-25cbc329667a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb874fe1-5610-449f-8c84-25cbc329667a-maas-api-xks-ca\") pod \"bb874fe1-5610-449f-8c84-25cbc329667a\" (UID: \"bb874fe1-5610-449f-8c84-25cbc329667a\") "
Nov 03 22:22:44.304460 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:44.304320    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb874fe1-5610-449f-8c84-25cbc329667a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb874fe1-5610-449f-8c84-25cbc329667a-maas-api-openshift-ca\") pod \"bb874fe1-5610-449f-8c84-25cbc329667a\" (UID: \"bb874fe1-5610-449f-8c84-25cbc329667a\") "
Nov 03 22:22:44.304460 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:44.304365    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb874fe1-5610-449f-8c84-25cbc329667a-kube-api-access-8rsdh\" (UniqueName: \"kubernetes.io/projected/bb874fe1-5610-449f-8c84-25cbc329667a-kube-api-access-8rsdh\") pod \"bb874fe1-5610-449f-8c84-25cbc329667a\" (UID: \"bb874fe1-5610-449f-8c84-25cbc329667a\") "
Nov 03 22:22:44.304733 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:44.304699    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb874fe1-5610-449f-8c84-25cbc329667a-maas-api-openshift-ca" pod "bb874fe1-5610-449f-8c84-25cbc329667a" (UID: "bb874fe1-5610-449f-8c84-25cbc329667a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:22:44.305472 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:44.305451    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"]
Nov 03 22:22:44.306593 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:44.306561    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb874fe1-5610-449f-8c84-25cbc329667a-maas-api-xks-ca" pod "bb874fe1-5610-449f-8c84-25cbc329667a" (UID: "bb874fe1-5610-449f-8c84-25cbc329667a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:22:44.306688 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:44.306672    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb874fe1-5610-449f-8c84-25cbc329667a-kube-api-access-8rsdh" pod "bb874fe1-5610-449f-8c84-25cbc329667a" (UID: "bb874fe1-5610-449f-8c84-25cbc329667a"). InnerVolumeSpecName "kube-api-access-8rsdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:44.354801 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:44.354727    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-q8m6z"]
Nov 03 22:22:44.405584 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:44.405542    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb874fe1-5610-449f-8c84-25cbc329667a-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:44.405584 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:44.405577    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb874fe1-5610-449f-8c84-25cbc329667a-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:44.405584 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:44.405587    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rsdh\" (UniqueName: \"kubernetes.io/projected/bb874fe1-5610-449f-8c84-25cbc329667a-kube-api-access-8rsdh\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:45.086668 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.086633    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp" event={"ID":"bb874fe1-5610-449f-8c84-25cbc329667a","Type":"ContainerDied","Data":"c46b5d93d5164f624f7ecc5caec4f84953d7e8785dc40cc68d9ef08c998c853f"}
Nov 03 22:22:45.087156 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.086695    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c46b5d93d5164f624f7ecc5caec4f84953d7e8785dc40cc68d9ef08c998c853f"
Nov 03 22:22:45.087156 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.086766    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp"
Nov 03 22:22:45.094383 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.094192    5053 generic.go:356] "Generic (PLEG): container finished" podID="d17d1dfe-8556-43d4-b67f-d0e628552f0b" containerID="cf529980f5c953275d110a322ade4ef906f85992c34846c724e61fa8ba713068" exitCode=0
Nov 03 22:22:45.094383 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.094253    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs" event={"ID":"d17d1dfe-8556-43d4-b67f-d0e628552f0b","Type":"ContainerDied","Data":"cf529980f5c953275d110a322ade4ef906f85992c34846c724e61fa8ba713068"}
Nov 03 22:22:45.094383 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.094309    5053 scope.go:122] "RemoveContainer" containerID="adc7e544a6d458cdd2ef0b533d07f79cdca60a608588c54144966b1b578bed41"
Nov 03 22:22:45.165765 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.165703    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 22:22:45.213926 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.213889    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-podinfo\") pod \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") "
Nov 03 22:22:45.214107 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.213969    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-envoy\") pod \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") "
Nov 03 22:22:45.214107 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.214062    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-token\" (UniqueName: \"kubernetes.io/projected/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-token\") pod \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") "
Nov 03 22:22:45.214107 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.214097    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-workload-certs\") pod \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") "
Nov 03 22:22:45.214305 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.214139    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-workload-socket\") pod \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") "
Nov 03 22:22:45.214305 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.214183    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istiod-ca-cert\") pod \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") "
Nov 03 22:22:45.214305 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.214221    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-credential-socket\") pod \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") "
Nov 03 22:22:45.214467 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.214315    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d17d1dfe-8556-43d4-b67f-d0e628552f0b-kube-api-access-5jbpl\" (UniqueName: \"kubernetes.io/projected/d17d1dfe-8556-43d4-b67f-d0e628552f0b-kube-api-access-5jbpl\") pod \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") "
Nov 03 22:22:45.214467 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.214359    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-data\") pod \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\" (UID: \"d17d1dfe-8556-43d4-b67f-d0e628552f0b\") "
Nov 03 22:22:45.214467 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.214430    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-workload-certs" pod "d17d1dfe-8556-43d4-b67f-d0e628552f0b" (UID: "d17d1dfe-8556-43d4-b67f-d0e628552f0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:45.214637 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.214456    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-workload-socket" pod "d17d1dfe-8556-43d4-b67f-d0e628552f0b" (UID: "d17d1dfe-8556-43d4-b67f-d0e628552f0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:45.214696 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.214657    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-data" pod "d17d1dfe-8556-43d4-b67f-d0e628552f0b" (UID: "d17d1dfe-8556-43d4-b67f-d0e628552f0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:45.214696 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.214672    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-credential-socket" pod "d17d1dfe-8556-43d4-b67f-d0e628552f0b" (UID: "d17d1dfe-8556-43d4-b67f-d0e628552f0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:45.214823 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.214804    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:45.214879 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.214862    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:45.214936 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.214879    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:45.214936 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.214892    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:45.215037 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.215003    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istiod-ca-cert" pod "d17d1dfe-8556-43d4-b67f-d0e628552f0b" (UID: "d17d1dfe-8556-43d4-b67f-d0e628552f0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:22:45.216590 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.216564    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-envoy" pod "d17d1dfe-8556-43d4-b67f-d0e628552f0b" (UID: "d17d1dfe-8556-43d4-b67f-d0e628552f0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:45.216725 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.216648    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d17d1dfe-8556-43d4-b67f-d0e628552f0b-kube-api-access-5jbpl" pod "d17d1dfe-8556-43d4-b67f-d0e628552f0b" (UID: "d17d1dfe-8556-43d4-b67f-d0e628552f0b"). InnerVolumeSpecName "kube-api-access-5jbpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:45.216863 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.216849    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-podinfo" pod "d17d1dfe-8556-43d4-b67f-d0e628552f0b" (UID: "d17d1dfe-8556-43d4-b67f-d0e628552f0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:22:45.216938 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.216923    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-token" pod "d17d1dfe-8556-43d4-b67f-d0e628552f0b" (UID: "d17d1dfe-8556-43d4-b67f-d0e628552f0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:45.316371 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.316335    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:45.316506 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.316377    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:45.316506 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.316390    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:45.316506 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.316404    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d17d1dfe-8556-43d4-b67f-d0e628552f0b-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:45.316506 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.316417    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jbpl\" (UniqueName: \"kubernetes.io/projected/d17d1dfe-8556-43d4-b67f-d0e628552f0b-kube-api-access-5jbpl\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:45.666829 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:45.666796    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9207d7a-c808-4e41-8e12-c3219c4f8215" path="/var/lib/kubelet/pods/f9207d7a-c808-4e41-8e12-c3219c4f8215/volumes"
Nov 03 22:22:46.101762 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:46.101731    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"
Nov 03 22:22:46.102286 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:46.101737    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs" event={"ID":"d17d1dfe-8556-43d4-b67f-d0e628552f0b","Type":"ContainerDied","Data":"7bed2da7de8ab50f1aed39bf03c0ff3f46c73af91fdb6e09246c297a8bbf1e3e"}
Nov 03 22:22:46.102286 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:46.101881    5053 scope.go:122] "RemoveContainer" containerID="cf529980f5c953275d110a322ade4ef906f85992c34846c724e61fa8ba713068"
Nov 03 22:22:46.317782 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:46.317745    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"]
Nov 03 22:22:46.323061 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:46.323019    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6ddbba-openshift-default-59d4b68dc5-hvczs"]
Nov 03 22:22:47.425283 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.425230    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns"]
Nov 03 22:22:47.425777 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.425747    5053 cpu_manager.go:414] "containerMap: removing container" podUID="d17d1dfe-8556-43d4-b67f-d0e628552f0b" containerName="istio-proxy"
Nov 03 22:22:47.425777 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.425759    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d17d1dfe-8556-43d4-b67f-d0e628552f0b" containerName="istio-proxy"
Nov 03 22:22:47.425777 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.425774    5053 cpu_manager.go:414] "containerMap: removing container" podUID="d17d1dfe-8556-43d4-b67f-d0e628552f0b" containerName="istio-proxy"
Nov 03 22:22:47.425777 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.425779    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d17d1dfe-8556-43d4-b67f-d0e628552f0b" containerName="istio-proxy"
Nov 03 22:22:47.425912 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.425795    5053 cpu_manager.go:414] "containerMap: removing container" podUID="f9207d7a-c808-4e41-8e12-c3219c4f8215" containerName="istio-proxy"
Nov 03 22:22:47.425912 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.425800    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9207d7a-c808-4e41-8e12-c3219c4f8215" containerName="istio-proxy"
Nov 03 22:22:47.425912 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.425816    5053 cpu_manager.go:414] "containerMap: removing container" podUID="bb874fe1-5610-449f-8c84-25cbc329667a" containerName="revoke-keys"
Nov 03 22:22:47.425912 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.425823    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb874fe1-5610-449f-8c84-25cbc329667a" containerName="revoke-keys"
Nov 03 22:22:47.425912 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.425911    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="d17d1dfe-8556-43d4-b67f-d0e628552f0b" containerName="istio-proxy"
Nov 03 22:22:47.426065 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.425921    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9207d7a-c808-4e41-8e12-c3219c4f8215" containerName="istio-proxy"
Nov 03 22:22:47.426065 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.425929    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb874fe1-5610-449f-8c84-25cbc329667a" containerName="revoke-keys"
Nov 03 22:22:47.430303 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.430285    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns"
Nov 03 22:22:47.485974 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.485938    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns"]
Nov 03 22:22:47.531492 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.531464    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:22:47.531492 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.531464    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:22:47.539995 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.539963    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a357ff2-aa9f-400f-bd60-b9ad7a1a714d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns\" (UID: \"1a357ff2-aa9f-400f-bd60-b9ad7a1a714d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns"
Nov 03 22:22:47.540174 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.540152    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4t6f\" (UniqueName: \"kubernetes.io/projected/1a357ff2-aa9f-400f-bd60-b9ad7a1a714d-kube-api-access-m4t6f\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns\" (UID: \"1a357ff2-aa9f-400f-bd60-b9ad7a1a714d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns"
Nov 03 22:22:47.540218 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.540187    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a357ff2-aa9f-400f-bd60-b9ad7a1a714d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns\" (UID: \"1a357ff2-aa9f-400f-bd60-b9ad7a1a714d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns"
Nov 03 22:22:47.641276 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.641206    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4t6f\" (UniqueName: \"kubernetes.io/projected/1a357ff2-aa9f-400f-bd60-b9ad7a1a714d-kube-api-access-m4t6f\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns\" (UID: \"1a357ff2-aa9f-400f-bd60-b9ad7a1a714d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns"
Nov 03 22:22:47.641276 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.641271    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a357ff2-aa9f-400f-bd60-b9ad7a1a714d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns\" (UID: \"1a357ff2-aa9f-400f-bd60-b9ad7a1a714d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns"
Nov 03 22:22:47.641579 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.641323    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a357ff2-aa9f-400f-bd60-b9ad7a1a714d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns\" (UID: \"1a357ff2-aa9f-400f-bd60-b9ad7a1a714d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns"
Nov 03 22:22:47.641947 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.641924    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a357ff2-aa9f-400f-bd60-b9ad7a1a714d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns\" (UID: \"1a357ff2-aa9f-400f-bd60-b9ad7a1a714d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns"
Nov 03 22:22:47.643817 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.643794    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a357ff2-aa9f-400f-bd60-b9ad7a1a714d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns\" (UID: \"1a357ff2-aa9f-400f-bd60-b9ad7a1a714d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns"
Nov 03 22:22:47.666635 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.666602    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d17d1dfe-8556-43d4-b67f-d0e628552f0b" path="/var/lib/kubelet/pods/d17d1dfe-8556-43d4-b67f-d0e628552f0b/volumes"
Nov 03 22:22:47.697971 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.697941    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4t6f\" (UniqueName: \"kubernetes.io/projected/1a357ff2-aa9f-400f-bd60-b9ad7a1a714d-kube-api-access-m4t6f\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns\" (UID: \"1a357ff2-aa9f-400f-bd60-b9ad7a1a714d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns"
Nov 03 22:22:47.743489 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:47.743453    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns"
Nov 03 22:22:48.115059 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:48.115025    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns"]
Nov 03 22:22:48.115897 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:22:48.115871    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a357ff2_aa9f_400f_bd60_b9ad7a1a714d.slice/crio-46fe6613c4ab4438be6095426ae8534490d911fb28987fcce6a706dd77dae3d2 WatchSource:0}: Error finding container 46fe6613c4ab4438be6095426ae8534490d911fb28987fcce6a706dd77dae3d2: Status 404 returned error can't find the container with id 46fe6613c4ab4438be6095426ae8534490d911fb28987fcce6a706dd77dae3d2
Nov 03 22:22:49.036422 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.036389    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b"
Nov 03 22:22:49.122271 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.122219    5053 generic.go:356] "Generic (PLEG): container finished" podID="1a357ff2-aa9f-400f-bd60-b9ad7a1a714d" containerID="9afcc2582c202cc7270a32772f700c801573e8bd9d61e9ce5ca34f53bc6b8c8a" exitCode=0
Nov 03 22:22:49.122271 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.122266    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns" event={"ID":"1a357ff2-aa9f-400f-bd60-b9ad7a1a714d","Type":"ContainerDied","Data":"9afcc2582c202cc7270a32772f700c801573e8bd9d61e9ce5ca34f53bc6b8c8a"}
Nov 03 22:22:49.122481 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.122313    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns" event={"ID":"1a357ff2-aa9f-400f-bd60-b9ad7a1a714d","Type":"ContainerStarted","Data":"46fe6613c4ab4438be6095426ae8534490d911fb28987fcce6a706dd77dae3d2"}
Nov 03 22:22:49.123735 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.123713    5053 generic.go:356] "Generic (PLEG): container finished" podID="b4036e26-b7be-4c14-bd68-4863da3c3fe0" containerID="dff7de5b02b5d5a738a6e8e311a95694d4761c80d6a740c6a35354ac743898b2" exitCode=0
Nov 03 22:22:49.123893 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.123766    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b"
Nov 03 22:22:49.123893 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.123801    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b" event={"ID":"b4036e26-b7be-4c14-bd68-4863da3c3fe0","Type":"ContainerDied","Data":"dff7de5b02b5d5a738a6e8e311a95694d4761c80d6a740c6a35354ac743898b2"}
Nov 03 22:22:49.123893 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.123851    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b" event={"ID":"b4036e26-b7be-4c14-bd68-4863da3c3fe0","Type":"ContainerDied","Data":"0e7f493c234b1074fbbc6cf0aa4dd365b825c35830f1502050d5e685553017bf"}
Nov 03 22:22:49.123893 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.123880    5053 scope.go:122] "RemoveContainer" containerID="dff7de5b02b5d5a738a6e8e311a95694d4761c80d6a740c6a35354ac743898b2"
Nov 03 22:22:49.125277 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.125256    5053 generic.go:356] "Generic (PLEG): container finished" podID="4cee4995-fb03-43bc-a63d-f764dba58a39" containerID="cd8dc383b439f79e13e223c48e54410069c13ff033541735d99b9980018e9fda" exitCode=0
Nov 03 22:22:49.125399 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.125293    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9" event={"ID":"4cee4995-fb03-43bc-a63d-f764dba58a39","Type":"ContainerDied","Data":"cd8dc383b439f79e13e223c48e54410069c13ff033541735d99b9980018e9fda"}
Nov 03 22:22:49.125399 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.125337    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9" event={"ID":"4cee4995-fb03-43bc-a63d-f764dba58a39","Type":"ContainerDied","Data":"5aaff74c73fe33b896c2521e6d0ee3f780693d726a88829afda134d38c21908c"}
Nov 03 22:22:49.125399 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.125361    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5aaff74c73fe33b896c2521e6d0ee3f780693d726a88829afda134d38c21908c"
Nov 03 22:22:49.134695 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.134664    5053 scope.go:122] "RemoveContainer" containerID="dff7de5b02b5d5a738a6e8e311a95694d4761c80d6a740c6a35354ac743898b2"
Nov 03 22:22:49.134947 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:22:49.134927    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff7de5b02b5d5a738a6e8e311a95694d4761c80d6a740c6a35354ac743898b2\": container with ID starting with dff7de5b02b5d5a738a6e8e311a95694d4761c80d6a740c6a35354ac743898b2 not found: ID does not exist" containerID="dff7de5b02b5d5a738a6e8e311a95694d4761c80d6a740c6a35354ac743898b2"
Nov 03 22:22:49.135009 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.134956    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff7de5b02b5d5a738a6e8e311a95694d4761c80d6a740c6a35354ac743898b2"} err="failed to get container status \"dff7de5b02b5d5a738a6e8e311a95694d4761c80d6a740c6a35354ac743898b2\": rpc error: code = NotFound desc = could not find container \"dff7de5b02b5d5a738a6e8e311a95694d4761c80d6a740c6a35354ac743898b2\": container with ID starting with dff7de5b02b5d5a738a6e8e311a95694d4761c80d6a740c6a35354ac743898b2 not found: ID does not exist"
Nov 03 22:22:49.135773 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.135758    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9"
Nov 03 22:22:49.158922 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.158889    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4036e26-b7be-4c14-bd68-4863da3c3fe0-kube-api-access-zwq5z\" (UniqueName: \"kubernetes.io/projected/b4036e26-b7be-4c14-bd68-4863da3c3fe0-kube-api-access-zwq5z\") pod \"b4036e26-b7be-4c14-bd68-4863da3c3fe0\" (UID: \"b4036e26-b7be-4c14-bd68-4863da3c3fe0\") "
Nov 03 22:22:49.159110 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.159094    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4036e26-b7be-4c14-bd68-4863da3c3fe0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4036e26-b7be-4c14-bd68-4863da3c3fe0-plugins-config-volume\") pod \"b4036e26-b7be-4c14-bd68-4863da3c3fe0\" (UID: \"b4036e26-b7be-4c14-bd68-4863da3c3fe0\") "
Nov 03 22:22:49.159547 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.159517    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4036e26-b7be-4c14-bd68-4863da3c3fe0-plugins-config-volume" pod "b4036e26-b7be-4c14-bd68-4863da3c3fe0" (UID: "b4036e26-b7be-4c14-bd68-4863da3c3fe0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:22:49.161277 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.161223    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4036e26-b7be-4c14-bd68-4863da3c3fe0-kube-api-access-zwq5z" pod "b4036e26-b7be-4c14-bd68-4863da3c3fe0" (UID: "b4036e26-b7be-4c14-bd68-4863da3c3fe0"). InnerVolumeSpecName "kube-api-access-zwq5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:49.259863 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.259826    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cee4995-fb03-43bc-a63d-f764dba58a39-kube-api-access-q5s85\" (UniqueName: \"kubernetes.io/projected/4cee4995-fb03-43bc-a63d-f764dba58a39-kube-api-access-q5s85\") pod \"4cee4995-fb03-43bc-a63d-f764dba58a39\" (UID: \"4cee4995-fb03-43bc-a63d-f764dba58a39\") "
Nov 03 22:22:49.260076 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.259877    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cee4995-fb03-43bc-a63d-f764dba58a39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cee4995-fb03-43bc-a63d-f764dba58a39-plugins-config-volume\") pod \"4cee4995-fb03-43bc-a63d-f764dba58a39\" (UID: \"4cee4995-fb03-43bc-a63d-f764dba58a39\") "
Nov 03 22:22:49.260221 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.260199    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4036e26-b7be-4c14-bd68-4863da3c3fe0-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:49.260221 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.260218    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwq5z\" (UniqueName: \"kubernetes.io/projected/b4036e26-b7be-4c14-bd68-4863da3c3fe0-kube-api-access-zwq5z\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:49.260413 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.260264    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cee4995-fb03-43bc-a63d-f764dba58a39-plugins-config-volume" pod "4cee4995-fb03-43bc-a63d-f764dba58a39" (UID: "4cee4995-fb03-43bc-a63d-f764dba58a39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:22:49.262133 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.262111    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cee4995-fb03-43bc-a63d-f764dba58a39-kube-api-access-q5s85" pod "4cee4995-fb03-43bc-a63d-f764dba58a39" (UID: "4cee4995-fb03-43bc-a63d-f764dba58a39"). InnerVolumeSpecName "kube-api-access-q5s85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:49.361451 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.361362    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5s85\" (UniqueName: \"kubernetes.io/projected/4cee4995-fb03-43bc-a63d-f764dba58a39-kube-api-access-q5s85\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:49.361451 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.361391    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cee4995-fb03-43bc-a63d-f764dba58a39-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:49.545847 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.545797    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b"]
Nov 03 22:22:49.548473 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.548434    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-6de8e6-6649b95f86-mvg9b"]
Nov 03 22:22:49.667369 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:49.667343    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4036e26-b7be-4c14-bd68-4863da3c3fe0" path="/var/lib/kubelet/pods/b4036e26-b7be-4c14-bd68-4863da3c3fe0/volumes"
Nov 03 22:22:50.132777 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:50.132746    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9"
Nov 03 22:22:50.279557 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:50.279528    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns"
Nov 03 22:22:50.300769 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:50.300737    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9"]
Nov 03 22:22:50.309087 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:50.309058    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6de8e6-68686d5bf9-tjxr9"]
Nov 03 22:22:50.371726 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:50.371695    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a357ff2-aa9f-400f-bd60-b9ad7a1a714d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a357ff2-aa9f-400f-bd60-b9ad7a1a714d-maas-api-openshift-ca\") pod \"1a357ff2-aa9f-400f-bd60-b9ad7a1a714d\" (UID: \"1a357ff2-aa9f-400f-bd60-b9ad7a1a714d\") "
Nov 03 22:22:50.371930 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:50.371746    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a357ff2-aa9f-400f-bd60-b9ad7a1a714d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a357ff2-aa9f-400f-bd60-b9ad7a1a714d-maas-api-xks-ca\") pod \"1a357ff2-aa9f-400f-bd60-b9ad7a1a714d\" (UID: \"1a357ff2-aa9f-400f-bd60-b9ad7a1a714d\") "
Nov 03 22:22:50.371930 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:50.371770    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a357ff2-aa9f-400f-bd60-b9ad7a1a714d-kube-api-access-m4t6f\" (UniqueName: \"kubernetes.io/projected/1a357ff2-aa9f-400f-bd60-b9ad7a1a714d-kube-api-access-m4t6f\") pod \"1a357ff2-aa9f-400f-bd60-b9ad7a1a714d\" (UID: \"1a357ff2-aa9f-400f-bd60-b9ad7a1a714d\") "
Nov 03 22:22:50.372144 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:50.372119    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a357ff2-aa9f-400f-bd60-b9ad7a1a714d-maas-api-openshift-ca" pod "1a357ff2-aa9f-400f-bd60-b9ad7a1a714d" (UID: "1a357ff2-aa9f-400f-bd60-b9ad7a1a714d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:22:50.374073 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:50.374048    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a357ff2-aa9f-400f-bd60-b9ad7a1a714d-maas-api-xks-ca" pod "1a357ff2-aa9f-400f-bd60-b9ad7a1a714d" (UID: "1a357ff2-aa9f-400f-bd60-b9ad7a1a714d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:22:50.374073 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:50.374064    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a357ff2-aa9f-400f-bd60-b9ad7a1a714d-kube-api-access-m4t6f" pod "1a357ff2-aa9f-400f-bd60-b9ad7a1a714d" (UID: "1a357ff2-aa9f-400f-bd60-b9ad7a1a714d"). InnerVolumeSpecName "kube-api-access-m4t6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:50.472807 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:50.472773    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a357ff2-aa9f-400f-bd60-b9ad7a1a714d-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:50.472807 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:50.472804    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a357ff2-aa9f-400f-bd60-b9ad7a1a714d-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:50.472807 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:50.472815    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4t6f\" (UniqueName: \"kubernetes.io/projected/1a357ff2-aa9f-400f-bd60-b9ad7a1a714d-kube-api-access-m4t6f\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:51.140075 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:51.140046    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns"
Nov 03 22:22:51.140512 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:51.140041    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns" event={"ID":"1a357ff2-aa9f-400f-bd60-b9ad7a1a714d","Type":"ContainerDied","Data":"46fe6613c4ab4438be6095426ae8534490d911fb28987fcce6a706dd77dae3d2"}
Nov 03 22:22:51.140512 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:51.140181    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46fe6613c4ab4438be6095426ae8534490d911fb28987fcce6a706dd77dae3d2"
Nov 03 22:22:51.666665 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:51.666633    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cee4995-fb03-43bc-a63d-f764dba58a39" path="/var/lib/kubelet/pods/4cee4995-fb03-43bc-a63d-f764dba58a39/volumes"
Nov 03 22:22:52.616505 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:52.616463    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr"]
Nov 03 22:22:54.309621 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.309584    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f9432e8-q62fh"]
Nov 03 22:22:54.310078 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.310065    5053 cpu_manager.go:414] "containerMap: removing container" podUID="f9207d7a-c808-4e41-8e12-c3219c4f8215" containerName="istio-proxy"
Nov 03 22:22:54.310145 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.310079    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9207d7a-c808-4e41-8e12-c3219c4f8215" containerName="istio-proxy"
Nov 03 22:22:54.310145 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.310088    5053 cpu_manager.go:414] "containerMap: removing container" podUID="1a357ff2-aa9f-400f-bd60-b9ad7a1a714d" containerName="revoke-keys"
Nov 03 22:22:54.310145 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.310094    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a357ff2-aa9f-400f-bd60-b9ad7a1a714d" containerName="revoke-keys"
Nov 03 22:22:54.310145 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.310117    5053 cpu_manager.go:414] "containerMap: removing container" podUID="4cee4995-fb03-43bc-a63d-f764dba58a39" containerName="payload-pre-processing"
Nov 03 22:22:54.310145 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.310123    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cee4995-fb03-43bc-a63d-f764dba58a39" containerName="payload-pre-processing"
Nov 03 22:22:54.310145 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.310142    5053 cpu_manager.go:414] "containerMap: removing container" podUID="b4036e26-b7be-4c14-bd68-4863da3c3fe0" containerName="payload-processing"
Nov 03 22:22:54.310394 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.310149    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4036e26-b7be-4c14-bd68-4863da3c3fe0" containerName="payload-processing"
Nov 03 22:22:54.310394 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.310219    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9207d7a-c808-4e41-8e12-c3219c4f8215" containerName="istio-proxy"
Nov 03 22:22:54.310394 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.310228    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a357ff2-aa9f-400f-bd60-b9ad7a1a714d" containerName="revoke-keys"
Nov 03 22:22:54.310394 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.310272    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cee4995-fb03-43bc-a63d-f764dba58a39" containerName="payload-pre-processing"
Nov 03 22:22:54.310394 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.310281    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4036e26-b7be-4c14-bd68-4863da3c3fe0" containerName="payload-processing"
Nov 03 22:22:54.310394 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.310289    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="d17d1dfe-8556-43d4-b67f-d0e628552f0b" containerName="istio-proxy"
Nov 03 22:22:54.313453 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.313436    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9432e8-q62fh"
Nov 03 22:22:54.367207 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.367174    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:22:54.414097 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.414058    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a65b845d-c808-431e-b18d-976a3f93604f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f9432e8-q62fh\" (UID: \"a65b845d-c808-431e-b18d-976a3f93604f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9432e8-q62fh"
Nov 03 22:22:54.414332 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.414111    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d45x\" (UniqueName: \"kubernetes.io/projected/a65b845d-c808-431e-b18d-976a3f93604f-kube-api-access-2d45x\") pod \"maas-api-revoke-sub-0f9432e8-q62fh\" (UID: \"a65b845d-c808-431e-b18d-976a3f93604f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9432e8-q62fh"
Nov 03 22:22:54.414332 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.414223    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a65b845d-c808-431e-b18d-976a3f93604f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f9432e8-q62fh\" (UID: \"a65b845d-c808-431e-b18d-976a3f93604f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9432e8-q62fh"
Nov 03 22:22:54.416149 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.416104    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:22:54.420331 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.420302    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f9432e8-q62fh"]
Nov 03 22:22:54.515208 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.515168    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a65b845d-c808-431e-b18d-976a3f93604f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f9432e8-q62fh\" (UID: \"a65b845d-c808-431e-b18d-976a3f93604f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9432e8-q62fh"
Nov 03 22:22:54.515208 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.515214    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d45x\" (UniqueName: \"kubernetes.io/projected/a65b845d-c808-431e-b18d-976a3f93604f-kube-api-access-2d45x\") pod \"maas-api-revoke-sub-0f9432e8-q62fh\" (UID: \"a65b845d-c808-431e-b18d-976a3f93604f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9432e8-q62fh"
Nov 03 22:22:54.515516 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.515270    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a65b845d-c808-431e-b18d-976a3f93604f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f9432e8-q62fh\" (UID: \"a65b845d-c808-431e-b18d-976a3f93604f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9432e8-q62fh"
Nov 03 22:22:54.515862 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.515839    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a65b845d-c808-431e-b18d-976a3f93604f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f9432e8-q62fh\" (UID: \"a65b845d-c808-431e-b18d-976a3f93604f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9432e8-q62fh"
Nov 03 22:22:54.518215 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.518193    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a65b845d-c808-431e-b18d-976a3f93604f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f9432e8-q62fh\" (UID: \"a65b845d-c808-431e-b18d-976a3f93604f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9432e8-q62fh"
Nov 03 22:22:54.572825 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.572750    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d45x\" (UniqueName: \"kubernetes.io/projected/a65b845d-c808-431e-b18d-976a3f93604f-kube-api-access-2d45x\") pod \"maas-api-revoke-sub-0f9432e8-q62fh\" (UID: \"a65b845d-c808-431e-b18d-976a3f93604f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9432e8-q62fh"
Nov 03 22:22:54.627814 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:54.627780    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9432e8-q62fh"
Nov 03 22:22:55.005640 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:55.005606    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f9432e8-q62fh"]
Nov 03 22:22:55.006866 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:22:55.006835    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65b845d_c808_431e_b18d_976a3f93604f.slice/crio-be802239d01578edb0d6ee446b8be91947f7df812fc091d2750fa59fca2b9131 WatchSource:0}: Error finding container be802239d01578edb0d6ee446b8be91947f7df812fc091d2750fa59fca2b9131: Status 404 returned error can't find the container with id be802239d01578edb0d6ee446b8be91947f7df812fc091d2750fa59fca2b9131
Nov 03 22:22:55.171734 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:55.171702    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9432e8-q62fh" event={"ID":"a65b845d-c808-431e-b18d-976a3f93604f","Type":"ContainerStarted","Data":"be802239d01578edb0d6ee446b8be91947f7df812fc091d2750fa59fca2b9131"}
Nov 03 22:22:56.179588 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:56.179503    5053 generic.go:356] "Generic (PLEG): container finished" podID="a65b845d-c808-431e-b18d-976a3f93604f" containerID="ad88da0fd610fb7102e899eeaf9ca4b4953c6a9d77adb9e06c4775e8306ece01" exitCode=0
Nov 03 22:22:56.179972 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:56.179591    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9432e8-q62fh" event={"ID":"a65b845d-c808-431e-b18d-976a3f93604f","Type":"ContainerDied","Data":"ad88da0fd610fb7102e899eeaf9ca4b4953c6a9d77adb9e06c4775e8306ece01"}
Nov 03 22:22:56.758952 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:56.758922    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn"]
Nov 03 22:22:56.763299 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:56.763274    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6de8e6-qxthn"]
Nov 03 22:22:57.341360 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:57.341324    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9432e8-q62fh"
Nov 03 22:22:57.447378 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:57.447345    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a65b845d-c808-431e-b18d-976a3f93604f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a65b845d-c808-431e-b18d-976a3f93604f-maas-api-openshift-ca\") pod \"a65b845d-c808-431e-b18d-976a3f93604f\" (UID: \"a65b845d-c808-431e-b18d-976a3f93604f\") "
Nov 03 22:22:57.447556 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:57.447471    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a65b845d-c808-431e-b18d-976a3f93604f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a65b845d-c808-431e-b18d-976a3f93604f-maas-api-xks-ca\") pod \"a65b845d-c808-431e-b18d-976a3f93604f\" (UID: \"a65b845d-c808-431e-b18d-976a3f93604f\") "
Nov 03 22:22:57.447556 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:57.447548    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a65b845d-c808-431e-b18d-976a3f93604f-kube-api-access-2d45x\" (UniqueName: \"kubernetes.io/projected/a65b845d-c808-431e-b18d-976a3f93604f-kube-api-access-2d45x\") pod \"a65b845d-c808-431e-b18d-976a3f93604f\" (UID: \"a65b845d-c808-431e-b18d-976a3f93604f\") "
Nov 03 22:22:57.447774 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:57.447748    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65b845d-c808-431e-b18d-976a3f93604f-maas-api-openshift-ca" pod "a65b845d-c808-431e-b18d-976a3f93604f" (UID: "a65b845d-c808-431e-b18d-976a3f93604f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:22:57.449790 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:57.449757    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a65b845d-c808-431e-b18d-976a3f93604f-maas-api-xks-ca" pod "a65b845d-c808-431e-b18d-976a3f93604f" (UID: "a65b845d-c808-431e-b18d-976a3f93604f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:22:57.449912 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:57.449855    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a65b845d-c808-431e-b18d-976a3f93604f-kube-api-access-2d45x" pod "a65b845d-c808-431e-b18d-976a3f93604f" (UID: "a65b845d-c808-431e-b18d-976a3f93604f"). InnerVolumeSpecName "kube-api-access-2d45x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:57.549081 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:57.549033    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a65b845d-c808-431e-b18d-976a3f93604f-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:57.549081 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:57.549071    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a65b845d-c808-431e-b18d-976a3f93604f-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:57.549081 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:57.549085    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d45x\" (UniqueName: \"kubernetes.io/projected/a65b845d-c808-431e-b18d-976a3f93604f-kube-api-access-2d45x\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:57.616839 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:57.616776    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr" podUID="ce8d8f89-c688-4928-8ea7-ec6b22a2c74e" containerName="payload-processing" containerID="cri-o://e4939359ac90512ce62e3059a98569684725aebb2f8d3fba23c1fbffbf111797" gracePeriod=25
Nov 03 22:22:57.666284 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:57.666250    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4201e070-91d5-445d-8eff-57bc5ef9ed00" path="/var/lib/kubelet/pods/4201e070-91d5-445d-8eff-57bc5ef9ed00/volumes"
Nov 03 22:22:58.194599 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:58.194558    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9432e8-q62fh" event={"ID":"a65b845d-c808-431e-b18d-976a3f93604f","Type":"ContainerDied","Data":"be802239d01578edb0d6ee446b8be91947f7df812fc091d2750fa59fca2b9131"}
Nov 03 22:22:58.194599 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:58.194576    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9432e8-q62fh"
Nov 03 22:22:58.194858 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:58.194744    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be802239d01578edb0d6ee446b8be91947f7df812fc091d2750fa59fca2b9131"
Nov 03 22:22:59.580806 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:22:59.580771    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr" podUID="ce8d8f89-c688-4928-8ea7-ec6b22a2c74e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:23:06.005197 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:06.005157    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"]
Nov 03 22:23:06.005662 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:06.005414    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf" podUID="bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518" containerName="maas-api" containerID="cri-o://e93a792cd716e63ad9e231e80debee828f7d8e219328b48c3daed07f13e63bf7" gracePeriod=30
Nov 03 22:23:06.244823 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:06.244787    5053 generic.go:356] "Generic (PLEG): container finished" podID="bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518" containerID="e93a792cd716e63ad9e231e80debee828f7d8e219328b48c3daed07f13e63bf7" exitCode=0
Nov 03 22:23:06.244956 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:06.244863    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf" event={"ID":"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518","Type":"ContainerDied","Data":"e93a792cd716e63ad9e231e80debee828f7d8e219328b48c3daed07f13e63bf7"}
Nov 03 22:23:06.244956 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:06.244916    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf" event={"ID":"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518","Type":"ContainerDied","Data":"a42e8bd5e72e1ea5d12440a7ce56a3c8bfa9f0fce23c53b6bc0a0fd5bc4b9e4a"}
Nov 03 22:23:06.244956 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:06.244935    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a42e8bd5e72e1ea5d12440a7ce56a3c8bfa9f0fce23c53b6bc0a0fd5bc4b9e4a"
Nov 03 22:23:06.259053 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:06.258995    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"
Nov 03 22:23:06.358071 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:06.358036    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-metrics-certs\") pod \"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518\" (UID: \"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518\") "
Nov 03 22:23:06.358281 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:06.358121    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-kube-api-access-tlx79\" (UniqueName: \"kubernetes.io/projected/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-kube-api-access-tlx79\") pod \"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518\" (UID: \"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518\") "
Nov 03 22:23:06.358281 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:06.358160    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-maas-api-tls\") pod \"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518\" (UID: \"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518\") "
Nov 03 22:23:06.358281 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:06.358219    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-openshift-service-ca\") pod \"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518\" (UID: \"bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518\") "
Nov 03 22:23:06.358627 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:06.358602    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-openshift-service-ca" pod "bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518" (UID: "bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:23:06.360454 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:06.360430    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-metrics-certs" pod "bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518" (UID: "bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:23:06.360454 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:06.360450    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-kube-api-access-tlx79" pod "bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518" (UID: "bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518"). InnerVolumeSpecName "kube-api-access-tlx79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:06.360580 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:06.360455    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-maas-api-tls" pod "bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518" (UID: "bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:23:06.459634 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:06.459598    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlx79\" (UniqueName: \"kubernetes.io/projected/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-kube-api-access-tlx79\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:06.459634 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:06.459631    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-maas-api-tls\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:06.459634 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:06.459643    5053 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-openshift-service-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:06.459886 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:06.459652    5053 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518-metrics-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:07.250998 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.250953    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"
Nov 03 22:23:07.252613 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.252588    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f514a6e-pc52f"]
Nov 03 22:23:07.253096 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.253079    5053 cpu_manager.go:414] "containerMap: removing container" podUID="bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518" containerName="maas-api"
Nov 03 22:23:07.253193 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.253098    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518" containerName="maas-api"
Nov 03 22:23:07.253193 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.253120    5053 cpu_manager.go:414] "containerMap: removing container" podUID="a65b845d-c808-431e-b18d-976a3f93604f" containerName="revoke-keys"
Nov 03 22:23:07.253193 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.253129    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a65b845d-c808-431e-b18d-976a3f93604f" containerName="revoke-keys"
Nov 03 22:23:07.253366 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.253336    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518" containerName="maas-api"
Nov 03 22:23:07.253366 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.253356    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="a65b845d-c808-431e-b18d-976a3f93604f" containerName="revoke-keys"
Nov 03 22:23:07.256786 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.256769    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f514a6e-pc52f"
Nov 03 22:23:07.268953 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.268928    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae36d0a-e7ff-4f6b-84b0-999559282e87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f514a6e-pc52f\" (UID: \"6ae36d0a-e7ff-4f6b-84b0-999559282e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f514a6e-pc52f"
Nov 03 22:23:07.269081 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.269010    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhrq9\" (UniqueName: \"kubernetes.io/projected/6ae36d0a-e7ff-4f6b-84b0-999559282e87-kube-api-access-zhrq9\") pod \"maas-api-revoke-sub-2f514a6e-pc52f\" (UID: \"6ae36d0a-e7ff-4f6b-84b0-999559282e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f514a6e-pc52f"
Nov 03 22:23:07.269128 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.269101    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae36d0a-e7ff-4f6b-84b0-999559282e87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f514a6e-pc52f\" (UID: \"6ae36d0a-e7ff-4f6b-84b0-999559282e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f514a6e-pc52f"
Nov 03 22:23:07.356299 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.356267    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 22:23:07.356519 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.356332    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 22:23:07.356519 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.356421    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:23:07.359103 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.359079    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27aa713a-7dbw4"]
Nov 03 22:23:07.362953 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.362935    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f514a6e-pc52f"]
Nov 03 22:23:07.363035 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.363025    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aa713a-7dbw4"
Nov 03 22:23:07.369552 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.369526    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27aa713a-7dbw4"]
Nov 03 22:23:07.370076 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.370048    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae36d0a-e7ff-4f6b-84b0-999559282e87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f514a6e-pc52f\" (UID: \"6ae36d0a-e7ff-4f6b-84b0-999559282e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f514a6e-pc52f"
Nov 03 22:23:07.370221 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.370169    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b43bd20d-e094-41be-8491-779f15085eb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27aa713a-7dbw4\" (UID: \"b43bd20d-e094-41be-8491-779f15085eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aa713a-7dbw4"
Nov 03 22:23:07.370312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.370249    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhrq9\" (UniqueName: \"kubernetes.io/projected/6ae36d0a-e7ff-4f6b-84b0-999559282e87-kube-api-access-zhrq9\") pod \"maas-api-revoke-sub-2f514a6e-pc52f\" (UID: \"6ae36d0a-e7ff-4f6b-84b0-999559282e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f514a6e-pc52f"
Nov 03 22:23:07.370312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.370306    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b43bd20d-e094-41be-8491-779f15085eb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27aa713a-7dbw4\" (UID: \"b43bd20d-e094-41be-8491-779f15085eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aa713a-7dbw4"
Nov 03 22:23:07.370434 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.370337    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhxdz\" (UniqueName: \"kubernetes.io/projected/b43bd20d-e094-41be-8491-779f15085eb5-kube-api-access-rhxdz\") pod \"maas-api-revoke-sub-27aa713a-7dbw4\" (UID: \"b43bd20d-e094-41be-8491-779f15085eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aa713a-7dbw4"
Nov 03 22:23:07.370434 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.370367    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae36d0a-e7ff-4f6b-84b0-999559282e87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f514a6e-pc52f\" (UID: \"6ae36d0a-e7ff-4f6b-84b0-999559282e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f514a6e-pc52f"
Nov 03 22:23:07.370867 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.370844    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae36d0a-e7ff-4f6b-84b0-999559282e87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f514a6e-pc52f\" (UID: \"6ae36d0a-e7ff-4f6b-84b0-999559282e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f514a6e-pc52f"
Nov 03 22:23:07.410868 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.410836    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:23:07.423637 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.423608    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae36d0a-e7ff-4f6b-84b0-999559282e87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f514a6e-pc52f\" (UID: \"6ae36d0a-e7ff-4f6b-84b0-999559282e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f514a6e-pc52f"
Nov 03 22:23:07.462066 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.462031    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhrq9\" (UniqueName: \"kubernetes.io/projected/6ae36d0a-e7ff-4f6b-84b0-999559282e87-kube-api-access-zhrq9\") pod \"maas-api-revoke-sub-2f514a6e-pc52f\" (UID: \"6ae36d0a-e7ff-4f6b-84b0-999559282e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f514a6e-pc52f"
Nov 03 22:23:07.471153 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.471109    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhxdz\" (UniqueName: \"kubernetes.io/projected/b43bd20d-e094-41be-8491-779f15085eb5-kube-api-access-rhxdz\") pod \"maas-api-revoke-sub-27aa713a-7dbw4\" (UID: \"b43bd20d-e094-41be-8491-779f15085eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aa713a-7dbw4"
Nov 03 22:23:07.471445 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.471423    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b43bd20d-e094-41be-8491-779f15085eb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27aa713a-7dbw4\" (UID: \"b43bd20d-e094-41be-8491-779f15085eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aa713a-7dbw4"
Nov 03 22:23:07.472058 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.471521    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b43bd20d-e094-41be-8491-779f15085eb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27aa713a-7dbw4\" (UID: \"b43bd20d-e094-41be-8491-779f15085eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aa713a-7dbw4"
Nov 03 22:23:07.472227 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.472205    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b43bd20d-e094-41be-8491-779f15085eb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27aa713a-7dbw4\" (UID: \"b43bd20d-e094-41be-8491-779f15085eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aa713a-7dbw4"
Nov 03 22:23:07.474301 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.474274    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b43bd20d-e094-41be-8491-779f15085eb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27aa713a-7dbw4\" (UID: \"b43bd20d-e094-41be-8491-779f15085eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aa713a-7dbw4"
Nov 03 22:23:07.528545 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.528461    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"]
Nov 03 22:23:07.530246 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.530213    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhxdz\" (UniqueName: \"kubernetes.io/projected/b43bd20d-e094-41be-8491-779f15085eb5-kube-api-access-rhxdz\") pod \"maas-api-revoke-sub-27aa713a-7dbw4\" (UID: \"b43bd20d-e094-41be-8491-779f15085eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aa713a-7dbw4"
Nov 03 22:23:07.531453 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.531429    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fb477-64cd77cf4-ggcrf"]
Nov 03 22:23:07.572046 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.572009    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f514a6e-pc52f"
Nov 03 22:23:07.669787 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.669754    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518" path="/var/lib/kubelet/pods/bbbd93ef-5bd8-4fdf-aecb-c425fe7a5518/volumes"
Nov 03 22:23:07.679799 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.679770    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aa713a-7dbw4"
Nov 03 22:23:07.970954 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:07.953286    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f514a6e-pc52f"]
Nov 03 22:23:07.970954 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:23:07.954451    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae36d0a_e7ff_4f6b_84b0_999559282e87.slice/crio-4e33ca1c4275045550c259f093e7bbb6bd95239d9eb46ea04b8d00a1430fc982 WatchSource:0}: Error finding container 4e33ca1c4275045550c259f093e7bbb6bd95239d9eb46ea04b8d00a1430fc982: Status 404 returned error can't find the container with id 4e33ca1c4275045550c259f093e7bbb6bd95239d9eb46ea04b8d00a1430fc982
Nov 03 22:23:08.062143 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:08.062108    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27aa713a-7dbw4"]
Nov 03 22:23:08.063986 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:23:08.063956    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb43bd20d_e094_41be_8491_779f15085eb5.slice/crio-581c667b253fec23a45e0e3eded95c3d1b1129866debd610bdb733be519aaf8e WatchSource:0}: Error finding container 581c667b253fec23a45e0e3eded95c3d1b1129866debd610bdb733be519aaf8e: Status 404 returned error can't find the container with id 581c667b253fec23a45e0e3eded95c3d1b1129866debd610bdb733be519aaf8e
Nov 03 22:23:08.220615 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:08.220593    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns"]
Nov 03 22:23:08.222812 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:08.222790    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fb477-c45ns"]
Nov 03 22:23:08.259107 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:08.259065    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f514a6e-pc52f" event={"ID":"6ae36d0a-e7ff-4f6b-84b0-999559282e87","Type":"ContainerStarted","Data":"4d1507ac2e82b00bdffaf92e7f4014f97484edadf30300fb2f440483d109da2d"}
Nov 03 22:23:08.259663 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:08.259123    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f514a6e-pc52f" event={"ID":"6ae36d0a-e7ff-4f6b-84b0-999559282e87","Type":"ContainerStarted","Data":"4e33ca1c4275045550c259f093e7bbb6bd95239d9eb46ea04b8d00a1430fc982"}
Nov 03 22:23:08.260395 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:08.260374    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aa713a-7dbw4" event={"ID":"b43bd20d-e094-41be-8491-779f15085eb5","Type":"ContainerStarted","Data":"581c667b253fec23a45e0e3eded95c3d1b1129866debd610bdb733be519aaf8e"}
Nov 03 22:23:08.376701 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:08.376644    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f514a6e-pc52f" podStartSLOduration=1.376629888 podStartE2EDuration="1.376629888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:08.374285134 +0000 UTC m=+4102.868207007" watchObservedRunningTime="2026-11-03 22:23:08.376629888 +0000 UTC m=+4102.870551763"
Nov 03 22:23:09.269055 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:09.269017    5053 generic.go:356] "Generic (PLEG): container finished" podID="6ae36d0a-e7ff-4f6b-84b0-999559282e87" containerID="4d1507ac2e82b00bdffaf92e7f4014f97484edadf30300fb2f440483d109da2d" exitCode=0
Nov 03 22:23:09.269532 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:09.269109    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f514a6e-pc52f" event={"ID":"6ae36d0a-e7ff-4f6b-84b0-999559282e87","Type":"ContainerDied","Data":"4d1507ac2e82b00bdffaf92e7f4014f97484edadf30300fb2f440483d109da2d"}
Nov 03 22:23:09.270933 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:09.270906    5053 generic.go:356] "Generic (PLEG): container finished" podID="b43bd20d-e094-41be-8491-779f15085eb5" containerID="5816220a42e2477bd45da78e40c16294ff691ced627fa43b5352b815ddc45c17" exitCode=0
Nov 03 22:23:09.271063 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:09.270985    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aa713a-7dbw4" event={"ID":"b43bd20d-e094-41be-8491-779f15085eb5","Type":"ContainerDied","Data":"5816220a42e2477bd45da78e40c16294ff691ced627fa43b5352b815ddc45c17"}
Nov 03 22:23:09.580754 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:09.580647    5053 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr" podUID="ce8d8f89-c688-4928-8ea7-ec6b22a2c74e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:23:09.667011 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:09.666980    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a357ff2-aa9f-400f-bd60-b9ad7a1a714d" path="/var/lib/kubelet/pods/1a357ff2-aa9f-400f-bd60-b9ad7a1a714d/volumes"
Nov 03 22:23:09.667427 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:09.667407    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h"]
Nov 03 22:23:09.669193 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:09.669175    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7ee809-7l85h"]
Nov 03 22:23:10.435872 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:10.435845    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aa713a-7dbw4"
Nov 03 22:23:10.454987 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:10.454958    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f514a6e-pc52f"
Nov 03 22:23:10.499948 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:10.499915    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b43bd20d-e094-41be-8491-779f15085eb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b43bd20d-e094-41be-8491-779f15085eb5-maas-api-xks-ca\") pod \"b43bd20d-e094-41be-8491-779f15085eb5\" (UID: \"b43bd20d-e094-41be-8491-779f15085eb5\") "
Nov 03 22:23:10.500136 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:10.499974    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b43bd20d-e094-41be-8491-779f15085eb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b43bd20d-e094-41be-8491-779f15085eb5-maas-api-openshift-ca\") pod \"b43bd20d-e094-41be-8491-779f15085eb5\" (UID: \"b43bd20d-e094-41be-8491-779f15085eb5\") "
Nov 03 22:23:10.500136 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:10.500008    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b43bd20d-e094-41be-8491-779f15085eb5-kube-api-access-rhxdz\" (UniqueName: \"kubernetes.io/projected/b43bd20d-e094-41be-8491-779f15085eb5-kube-api-access-rhxdz\") pod \"b43bd20d-e094-41be-8491-779f15085eb5\" (UID: \"b43bd20d-e094-41be-8491-779f15085eb5\") "
Nov 03 22:23:10.500136 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:10.500042    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae36d0a-e7ff-4f6b-84b0-999559282e87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae36d0a-e7ff-4f6b-84b0-999559282e87-maas-api-openshift-ca\") pod \"6ae36d0a-e7ff-4f6b-84b0-999559282e87\" (UID: \"6ae36d0a-e7ff-4f6b-84b0-999559282e87\") "
Nov 03 22:23:10.500136 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:10.500064    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ae36d0a-e7ff-4f6b-84b0-999559282e87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae36d0a-e7ff-4f6b-84b0-999559282e87-maas-api-xks-ca\") pod \"6ae36d0a-e7ff-4f6b-84b0-999559282e87\" (UID: \"6ae36d0a-e7ff-4f6b-84b0-999559282e87\") "
Nov 03 22:23:10.500406 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:10.500157    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae36d0a-e7ff-4f6b-84b0-999559282e87-kube-api-access-zhrq9\" (UniqueName: \"kubernetes.io/projected/6ae36d0a-e7ff-4f6b-84b0-999559282e87-kube-api-access-zhrq9\") pod \"6ae36d0a-e7ff-4f6b-84b0-999559282e87\" (UID: \"6ae36d0a-e7ff-4f6b-84b0-999559282e87\") "
Nov 03 22:23:10.500468 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:10.500399    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b43bd20d-e094-41be-8491-779f15085eb5-maas-api-openshift-ca" pod "b43bd20d-e094-41be-8491-779f15085eb5" (UID: "b43bd20d-e094-41be-8491-779f15085eb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:23:10.500468 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:10.500424    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae36d0a-e7ff-4f6b-84b0-999559282e87-maas-api-openshift-ca" pod "6ae36d0a-e7ff-4f6b-84b0-999559282e87" (UID: "6ae36d0a-e7ff-4f6b-84b0-999559282e87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:23:10.500610 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:10.500586    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b43bd20d-e094-41be-8491-779f15085eb5-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:10.500910 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:10.500613    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae36d0a-e7ff-4f6b-84b0-999559282e87-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:10.502575 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:10.502536    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ae36d0a-e7ff-4f6b-84b0-999559282e87-maas-api-xks-ca" pod "6ae36d0a-e7ff-4f6b-84b0-999559282e87" (UID: "6ae36d0a-e7ff-4f6b-84b0-999559282e87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:23:10.502694 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:10.502618    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b43bd20d-e094-41be-8491-779f15085eb5-maas-api-xks-ca" pod "b43bd20d-e094-41be-8491-779f15085eb5" (UID: "b43bd20d-e094-41be-8491-779f15085eb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:23:10.502694 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:10.502640    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b43bd20d-e094-41be-8491-779f15085eb5-kube-api-access-rhxdz" pod "b43bd20d-e094-41be-8491-779f15085eb5" (UID: "b43bd20d-e094-41be-8491-779f15085eb5"). InnerVolumeSpecName "kube-api-access-rhxdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:10.502909 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:10.502893    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae36d0a-e7ff-4f6b-84b0-999559282e87-kube-api-access-zhrq9" pod "6ae36d0a-e7ff-4f6b-84b0-999559282e87" (UID: "6ae36d0a-e7ff-4f6b-84b0-999559282e87"). InnerVolumeSpecName "kube-api-access-zhrq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:10.602120 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:10.602024    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhrq9\" (UniqueName: \"kubernetes.io/projected/6ae36d0a-e7ff-4f6b-84b0-999559282e87-kube-api-access-zhrq9\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:10.602120 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:10.602054    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b43bd20d-e094-41be-8491-779f15085eb5-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:10.602120 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:10.602064    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhxdz\" (UniqueName: \"kubernetes.io/projected/b43bd20d-e094-41be-8491-779f15085eb5-kube-api-access-rhxdz\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:10.602120 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:10.602073    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae36d0a-e7ff-4f6b-84b0-999559282e87-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:10.928862 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:23:10.928777    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:23:11.286204 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:11.286179    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aa713a-7dbw4"
Nov 03 22:23:11.286408 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:11.286215    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aa713a-7dbw4" event={"ID":"b43bd20d-e094-41be-8491-779f15085eb5","Type":"ContainerDied","Data":"581c667b253fec23a45e0e3eded95c3d1b1129866debd610bdb733be519aaf8e"}
Nov 03 22:23:11.286408 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:11.286274    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="581c667b253fec23a45e0e3eded95c3d1b1129866debd610bdb733be519aaf8e"
Nov 03 22:23:11.287624 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:11.287595    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f514a6e-pc52f" event={"ID":"6ae36d0a-e7ff-4f6b-84b0-999559282e87","Type":"ContainerDied","Data":"4e33ca1c4275045550c259f093e7bbb6bd95239d9eb46ea04b8d00a1430fc982"}
Nov 03 22:23:11.287624 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:11.287618    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f514a6e-pc52f"
Nov 03 22:23:11.287789 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:11.287639    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e33ca1c4275045550c259f093e7bbb6bd95239d9eb46ea04b8d00a1430fc982"
Nov 03 22:23:11.666229 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:11.666147    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4ad58fc-198d-4b28-9f7a-380f56bc6257" path="/var/lib/kubelet/pods/f4ad58fc-198d-4b28-9f7a-380f56bc6257/volumes"
Nov 03 22:23:16.041727 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:16.041679    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj"]
Nov 03 22:23:16.042460 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:16.042440    5053 cpu_manager.go:414] "containerMap: removing container" podUID="6ae36d0a-e7ff-4f6b-84b0-999559282e87" containerName="revoke-keys"
Nov 03 22:23:16.042517 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:16.042464    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae36d0a-e7ff-4f6b-84b0-999559282e87" containerName="revoke-keys"
Nov 03 22:23:16.042517 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:16.042499    5053 cpu_manager.go:414] "containerMap: removing container" podUID="b43bd20d-e094-41be-8491-779f15085eb5" containerName="revoke-keys"
Nov 03 22:23:16.042517 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:16.042508    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b43bd20d-e094-41be-8491-779f15085eb5" containerName="revoke-keys"
Nov 03 22:23:16.042621 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:16.042611    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="b43bd20d-e094-41be-8491-779f15085eb5" containerName="revoke-keys"
Nov 03 22:23:16.042656 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:16.042633    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae36d0a-e7ff-4f6b-84b0-999559282e87" containerName="revoke-keys"
Nov 03 22:23:16.047780 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:16.047758    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj"
Nov 03 22:23:16.100289 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:16.100221    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj"]
Nov 03 22:23:16.146893 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:16.146856    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 22:23:16.146893 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:16.146872    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:23:16.146893 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:16.146873    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 22:23:16.148907 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:16.148886    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:23:16.160510 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:16.160482    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmrcm\" (UniqueName: \"kubernetes.io/projected/c5f25042-c855-47f7-af24-5c3818afaf80-kube-api-access-fmrcm\") pod \"maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj\" (UID: \"c5f25042-c855-47f7-af24-5c3818afaf80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj"
Nov 03 22:23:16.160647 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:16.160532    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f25042-c855-47f7-af24-5c3818afaf80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj\" (UID: \"c5f25042-c855-47f7-af24-5c3818afaf80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj"
Nov 03 22:23:16.160696 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:16.160668    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f25042-c855-47f7-af24-5c3818afaf80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj\" (UID: \"c5f25042-c855-47f7-af24-5c3818afaf80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj"
Nov 03 22:23:16.261455 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:16.261415    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f25042-c855-47f7-af24-5c3818afaf80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj\" (UID: \"c5f25042-c855-47f7-af24-5c3818afaf80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj"
Nov 03 22:23:16.261633 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:16.261479    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmrcm\" (UniqueName: \"kubernetes.io/projected/c5f25042-c855-47f7-af24-5c3818afaf80-kube-api-access-fmrcm\") pod \"maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj\" (UID: \"c5f25042-c855-47f7-af24-5c3818afaf80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj"
Nov 03 22:23:16.261633 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:16.261516    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f25042-c855-47f7-af24-5c3818afaf80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj\" (UID: \"c5f25042-c855-47f7-af24-5c3818afaf80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj"
Nov 03 22:23:16.262109 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:16.262082    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f25042-c855-47f7-af24-5c3818afaf80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj\" (UID: \"c5f25042-c855-47f7-af24-5c3818afaf80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj"
Nov 03 22:23:16.264052 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:16.264028    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f25042-c855-47f7-af24-5c3818afaf80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj\" (UID: \"c5f25042-c855-47f7-af24-5c3818afaf80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj"
Nov 03 22:23:16.319277 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:16.319157    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmrcm\" (UniqueName: \"kubernetes.io/projected/c5f25042-c855-47f7-af24-5c3818afaf80-kube-api-access-fmrcm\") pod \"maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj\" (UID: \"c5f25042-c855-47f7-af24-5c3818afaf80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj"
Nov 03 22:23:16.362052 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:16.362016    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj"
Nov 03 22:23:16.738085 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:16.738057    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj"]
Nov 03 22:23:16.740158 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:23:16.740132    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f25042_c855_47f7_af24_5c3818afaf80.slice/crio-750eba7e8540869952fafa9d736a22f080ece4513a89b923fb8dea96c1ed22de WatchSource:0}: Error finding container 750eba7e8540869952fafa9d736a22f080ece4513a89b923fb8dea96c1ed22de: Status 404 returned error can't find the container with id 750eba7e8540869952fafa9d736a22f080ece4513a89b923fb8dea96c1ed22de
Nov 03 22:23:17.329500 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:17.329403    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj" event={"ID":"c5f25042-c855-47f7-af24-5c3818afaf80","Type":"ContainerStarted","Data":"696b38268b46a9ef4c700920de52c1acf82fbd19cb3708f8da147062d8f04c17"}
Nov 03 22:23:17.329500 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:17.329483    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj" event={"ID":"c5f25042-c855-47f7-af24-5c3818afaf80","Type":"ContainerStarted","Data":"750eba7e8540869952fafa9d736a22f080ece4513a89b923fb8dea96c1ed22de"}
Nov 03 22:23:17.442764 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:17.442712    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj" podStartSLOduration=1.442697767 podStartE2EDuration="1.442697767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:17.441216411 +0000 UTC m=+4111.935138295" watchObservedRunningTime="2026-11-03 22:23:17.442697767 +0000 UTC m=+4111.936619644"
Nov 03 22:23:17.877312 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:17.877278    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr"
Nov 03 22:23:17.980550 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:17.980516    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce8d8f89-c688-4928-8ea7-ec6b22a2c74e-kube-api-access-hkncd\" (UniqueName: \"kubernetes.io/projected/ce8d8f89-c688-4928-8ea7-ec6b22a2c74e-kube-api-access-hkncd\") pod \"ce8d8f89-c688-4928-8ea7-ec6b22a2c74e\" (UID: \"ce8d8f89-c688-4928-8ea7-ec6b22a2c74e\") "
Nov 03 22:23:17.980750 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:17.980627    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce8d8f89-c688-4928-8ea7-ec6b22a2c74e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8d8f89-c688-4928-8ea7-ec6b22a2c74e-plugins-config-volume\") pod \"ce8d8f89-c688-4928-8ea7-ec6b22a2c74e\" (UID: \"ce8d8f89-c688-4928-8ea7-ec6b22a2c74e\") "
Nov 03 22:23:17.981011 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:17.980985    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce8d8f89-c688-4928-8ea7-ec6b22a2c74e-plugins-config-volume" pod "ce8d8f89-c688-4928-8ea7-ec6b22a2c74e" (UID: "ce8d8f89-c688-4928-8ea7-ec6b22a2c74e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:23:17.982886 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:17.982864    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce8d8f89-c688-4928-8ea7-ec6b22a2c74e-kube-api-access-hkncd" pod "ce8d8f89-c688-4928-8ea7-ec6b22a2c74e" (UID: "ce8d8f89-c688-4928-8ea7-ec6b22a2c74e"). InnerVolumeSpecName "kube-api-access-hkncd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:18.081830 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:18.081794    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkncd\" (UniqueName: \"kubernetes.io/projected/ce8d8f89-c688-4928-8ea7-ec6b22a2c74e-kube-api-access-hkncd\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:18.081830 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:18.081826    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8d8f89-c688-4928-8ea7-ec6b22a2c74e-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:18.337375 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:18.337339    5053 generic.go:356] "Generic (PLEG): container finished" podID="ce8d8f89-c688-4928-8ea7-ec6b22a2c74e" containerID="e4939359ac90512ce62e3059a98569684725aebb2f8d3fba23c1fbffbf111797" exitCode=0
Nov 03 22:23:18.337956 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:18.337384    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr"
Nov 03 22:23:18.337956 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:18.337431    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr" event={"ID":"ce8d8f89-c688-4928-8ea7-ec6b22a2c74e","Type":"ContainerDied","Data":"e4939359ac90512ce62e3059a98569684725aebb2f8d3fba23c1fbffbf111797"}
Nov 03 22:23:18.337956 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:18.337490    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr" event={"ID":"ce8d8f89-c688-4928-8ea7-ec6b22a2c74e","Type":"ContainerDied","Data":"7473c4c1a7d12a8e8fb20b72cea32310a3a402cd3a09ef0c4f9c8d14295642ec"}
Nov 03 22:23:18.337956 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:18.337521    5053 scope.go:122] "RemoveContainer" containerID="e4939359ac90512ce62e3059a98569684725aebb2f8d3fba23c1fbffbf111797"
Nov 03 22:23:18.339542 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:18.339516    5053 generic.go:356] "Generic (PLEG): container finished" podID="c5f25042-c855-47f7-af24-5c3818afaf80" containerID="696b38268b46a9ef4c700920de52c1acf82fbd19cb3708f8da147062d8f04c17" exitCode=0
Nov 03 22:23:18.339659 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:18.339580    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj" event={"ID":"c5f25042-c855-47f7-af24-5c3818afaf80","Type":"ContainerDied","Data":"696b38268b46a9ef4c700920de52c1acf82fbd19cb3708f8da147062d8f04c17"}
Nov 03 22:23:18.348469 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:18.348450    5053 scope.go:122] "RemoveContainer" containerID="e4939359ac90512ce62e3059a98569684725aebb2f8d3fba23c1fbffbf111797"
Nov 03 22:23:18.348756 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:23:18.348735    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4939359ac90512ce62e3059a98569684725aebb2f8d3fba23c1fbffbf111797\": container with ID starting with e4939359ac90512ce62e3059a98569684725aebb2f8d3fba23c1fbffbf111797 not found: ID does not exist" containerID="e4939359ac90512ce62e3059a98569684725aebb2f8d3fba23c1fbffbf111797"
Nov 03 22:23:18.348824 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:18.348769    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4939359ac90512ce62e3059a98569684725aebb2f8d3fba23c1fbffbf111797"} err="failed to get container status \"e4939359ac90512ce62e3059a98569684725aebb2f8d3fba23c1fbffbf111797\": rpc error: code = NotFound desc = could not find container \"e4939359ac90512ce62e3059a98569684725aebb2f8d3fba23c1fbffbf111797\": container with ID starting with e4939359ac90512ce62e3059a98569684725aebb2f8d3fba23c1fbffbf111797 not found: ID does not exist"
Nov 03 22:23:18.612365 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:18.612279    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr"]
Nov 03 22:23:18.617619 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:18.617593    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d71a74-65dd8dbd5d-xsxkr"]
Nov 03 22:23:19.495880 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:19.495847    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj"
Nov 03 22:23:19.597050 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:19.597012    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5f25042-c855-47f7-af24-5c3818afaf80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f25042-c855-47f7-af24-5c3818afaf80-maas-api-xks-ca\") pod \"c5f25042-c855-47f7-af24-5c3818afaf80\" (UID: \"c5f25042-c855-47f7-af24-5c3818afaf80\") "
Nov 03 22:23:19.597275 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:19.597069    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5f25042-c855-47f7-af24-5c3818afaf80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f25042-c855-47f7-af24-5c3818afaf80-maas-api-openshift-ca\") pod \"c5f25042-c855-47f7-af24-5c3818afaf80\" (UID: \"c5f25042-c855-47f7-af24-5c3818afaf80\") "
Nov 03 22:23:19.597275 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:19.597119    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5f25042-c855-47f7-af24-5c3818afaf80-kube-api-access-fmrcm\" (UniqueName: \"kubernetes.io/projected/c5f25042-c855-47f7-af24-5c3818afaf80-kube-api-access-fmrcm\") pod \"c5f25042-c855-47f7-af24-5c3818afaf80\" (UID: \"c5f25042-c855-47f7-af24-5c3818afaf80\") "
Nov 03 22:23:19.597544 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:19.597486    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5f25042-c855-47f7-af24-5c3818afaf80-maas-api-openshift-ca" pod "c5f25042-c855-47f7-af24-5c3818afaf80" (UID: "c5f25042-c855-47f7-af24-5c3818afaf80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:23:19.599433 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:19.599400    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f25042-c855-47f7-af24-5c3818afaf80-kube-api-access-fmrcm" pod "c5f25042-c855-47f7-af24-5c3818afaf80" (UID: "c5f25042-c855-47f7-af24-5c3818afaf80"). InnerVolumeSpecName "kube-api-access-fmrcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:19.599555 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:19.599533    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5f25042-c855-47f7-af24-5c3818afaf80-maas-api-xks-ca" pod "c5f25042-c855-47f7-af24-5c3818afaf80" (UID: "c5f25042-c855-47f7-af24-5c3818afaf80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:23:19.667452 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:19.667419    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce8d8f89-c688-4928-8ea7-ec6b22a2c74e" path="/var/lib/kubelet/pods/ce8d8f89-c688-4928-8ea7-ec6b22a2c74e/volumes"
Nov 03 22:23:19.698654 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:19.698606    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f25042-c855-47f7-af24-5c3818afaf80-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:19.698654 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:19.698643    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f25042-c855-47f7-af24-5c3818afaf80-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:19.698654 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:19.698659    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmrcm\" (UniqueName: \"kubernetes.io/projected/c5f25042-c855-47f7-af24-5c3818afaf80-kube-api-access-fmrcm\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:20.353597 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:20.353568    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj"
Nov 03 22:23:20.353787 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:20.353601    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj" event={"ID":"c5f25042-c855-47f7-af24-5c3818afaf80","Type":"ContainerDied","Data":"750eba7e8540869952fafa9d736a22f080ece4513a89b923fb8dea96c1ed22de"}
Nov 03 22:23:20.353787 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:20.353647    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="750eba7e8540869952fafa9d736a22f080ece4513a89b923fb8dea96c1ed22de"
Nov 03 22:23:20.523304 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:20.523269    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"]
Nov 03 22:23:21.405354 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:21.405318    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp"]
Nov 03 22:23:21.454442 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:21.454408    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d71a74-fzctp"]
Nov 03 22:23:21.666535 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:21.666452    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb874fe1-5610-449f-8c84-25cbc329667a" path="/var/lib/kubelet/pods/bb874fe1-5610-449f-8c84-25cbc329667a/volumes"
Nov 03 22:23:27.546342 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:27.546306    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt"]
Nov 03 22:23:27.651044 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:27.651002    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h"]
Nov 03 22:23:28.372104 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:28.372068    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"]
Nov 03 22:23:28.372636 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:28.372581    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj" podUID="88e0381b-b02d-40f1-acef-babc6c0248d1" containerName="istio-proxy" containerID="cri-o://2b600b493165ff5ed7b542af6da27f090da32cbcbd1c0605b93ccec390622bc8" gracePeriod=30
Nov 03 22:23:32.548116 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:32.548050    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt" podUID="770bdb59-84e0-449d-aaea-d75986b816b1" containerName="payload-processing" containerID="cri-o://d4b5ee122224e6d49d9925cc73b552008c719aa2003eaa1ca48afdf233d32bae" gracePeriod=25
Nov 03 22:23:32.652105 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:32.652059    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h" podUID="40e7f3d6-f0f4-456e-864e-1c3224bb3463" containerName="payload-pre-processing" containerID="cri-o://f5b5c9e2ddfec1c0e0aa0ac52337d096136207eb50c0fbfccfc672afa1e8a28a" gracePeriod=25
Nov 03 22:23:32.825965 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:32.825939    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt"
Nov 03 22:23:32.905621 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:32.905594    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h"
Nov 03 22:23:32.939425 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:32.939398    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/770bdb59-84e0-449d-aaea-d75986b816b1-kube-api-access-rht6m\" (UniqueName: \"kubernetes.io/projected/770bdb59-84e0-449d-aaea-d75986b816b1-kube-api-access-rht6m\") pod \"770bdb59-84e0-449d-aaea-d75986b816b1\" (UID: \"770bdb59-84e0-449d-aaea-d75986b816b1\") "
Nov 03 22:23:32.939613 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:32.939444    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/770bdb59-84e0-449d-aaea-d75986b816b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/770bdb59-84e0-449d-aaea-d75986b816b1-plugins-config-volume\") pod \"770bdb59-84e0-449d-aaea-d75986b816b1\" (UID: \"770bdb59-84e0-449d-aaea-d75986b816b1\") "
Nov 03 22:23:32.939865 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:32.939834    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/770bdb59-84e0-449d-aaea-d75986b816b1-plugins-config-volume" pod "770bdb59-84e0-449d-aaea-d75986b816b1" (UID: "770bdb59-84e0-449d-aaea-d75986b816b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:23:32.941997 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:32.941968    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770bdb59-84e0-449d-aaea-d75986b816b1-kube-api-access-rht6m" pod "770bdb59-84e0-449d-aaea-d75986b816b1" (UID: "770bdb59-84e0-449d-aaea-d75986b816b1"). InnerVolumeSpecName "kube-api-access-rht6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:33.040949 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.040917    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e7f3d6-f0f4-456e-864e-1c3224bb3463-kube-api-access-77tjr\" (UniqueName: \"kubernetes.io/projected/40e7f3d6-f0f4-456e-864e-1c3224bb3463-kube-api-access-77tjr\") pod \"40e7f3d6-f0f4-456e-864e-1c3224bb3463\" (UID: \"40e7f3d6-f0f4-456e-864e-1c3224bb3463\") "
Nov 03 22:23:33.041175 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.040963    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40e7f3d6-f0f4-456e-864e-1c3224bb3463-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e7f3d6-f0f4-456e-864e-1c3224bb3463-plugins-config-volume\") pod \"40e7f3d6-f0f4-456e-864e-1c3224bb3463\" (UID: \"40e7f3d6-f0f4-456e-864e-1c3224bb3463\") "
Nov 03 22:23:33.041230 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.041196    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rht6m\" (UniqueName: \"kubernetes.io/projected/770bdb59-84e0-449d-aaea-d75986b816b1-kube-api-access-rht6m\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:33.041230 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.041209    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/770bdb59-84e0-449d-aaea-d75986b816b1-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:33.041414 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.041390    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e7f3d6-f0f4-456e-864e-1c3224bb3463-plugins-config-volume" pod "40e7f3d6-f0f4-456e-864e-1c3224bb3463" (UID: "40e7f3d6-f0f4-456e-864e-1c3224bb3463"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:23:33.043227 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.043209    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e7f3d6-f0f4-456e-864e-1c3224bb3463-kube-api-access-77tjr" pod "40e7f3d6-f0f4-456e-864e-1c3224bb3463" (UID: "40e7f3d6-f0f4-456e-864e-1c3224bb3463"). InnerVolumeSpecName "kube-api-access-77tjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:33.142105 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.142025    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77tjr\" (UniqueName: \"kubernetes.io/projected/40e7f3d6-f0f4-456e-864e-1c3224bb3463-kube-api-access-77tjr\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:33.142105 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.142053    5053 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e7f3d6-f0f4-456e-864e-1c3224bb3463-plugins-config-volume\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:33.428983 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.428896    5053 generic.go:356] "Generic (PLEG): container finished" podID="770bdb59-84e0-449d-aaea-d75986b816b1" containerID="d4b5ee122224e6d49d9925cc73b552008c719aa2003eaa1ca48afdf233d32bae" exitCode=0
Nov 03 22:23:33.428983 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.428953    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt"
Nov 03 22:23:33.429248 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.428986    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt" event={"ID":"770bdb59-84e0-449d-aaea-d75986b816b1","Type":"ContainerDied","Data":"d4b5ee122224e6d49d9925cc73b552008c719aa2003eaa1ca48afdf233d32bae"}
Nov 03 22:23:33.429248 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.429065    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt" event={"ID":"770bdb59-84e0-449d-aaea-d75986b816b1","Type":"ContainerDied","Data":"9151e04fafe2c7c948ae859e094da167c3773413aaba9fadfa4bb093ca8503d5"}
Nov 03 22:23:33.429248 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.429095    5053 scope.go:122] "RemoveContainer" containerID="d4b5ee122224e6d49d9925cc73b552008c719aa2003eaa1ca48afdf233d32bae"
Nov 03 22:23:33.430444 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.430422    5053 generic.go:356] "Generic (PLEG): container finished" podID="40e7f3d6-f0f4-456e-864e-1c3224bb3463" containerID="f5b5c9e2ddfec1c0e0aa0ac52337d096136207eb50c0fbfccfc672afa1e8a28a" exitCode=0
Nov 03 22:23:33.430542 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.430469    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h"
Nov 03 22:23:33.430542 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.430496    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h" event={"ID":"40e7f3d6-f0f4-456e-864e-1c3224bb3463","Type":"ContainerDied","Data":"f5b5c9e2ddfec1c0e0aa0ac52337d096136207eb50c0fbfccfc672afa1e8a28a"}
Nov 03 22:23:33.430542 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.430532    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h" event={"ID":"40e7f3d6-f0f4-456e-864e-1c3224bb3463","Type":"ContainerDied","Data":"42ea98cb96b5fea5608decc81744f75385af5d7fa9f5dbbeabf5b25d5e2441fe"}
Nov 03 22:23:33.488038 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.488012    5053 scope.go:122] "RemoveContainer" containerID="d4b5ee122224e6d49d9925cc73b552008c719aa2003eaa1ca48afdf233d32bae"
Nov 03 22:23:33.488370 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:23:33.488346    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4b5ee122224e6d49d9925cc73b552008c719aa2003eaa1ca48afdf233d32bae\": container with ID starting with d4b5ee122224e6d49d9925cc73b552008c719aa2003eaa1ca48afdf233d32bae not found: ID does not exist" containerID="d4b5ee122224e6d49d9925cc73b552008c719aa2003eaa1ca48afdf233d32bae"
Nov 03 22:23:33.488481 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.488383    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4b5ee122224e6d49d9925cc73b552008c719aa2003eaa1ca48afdf233d32bae"} err="failed to get container status \"d4b5ee122224e6d49d9925cc73b552008c719aa2003eaa1ca48afdf233d32bae\": rpc error: code = NotFound desc = could not find container \"d4b5ee122224e6d49d9925cc73b552008c719aa2003eaa1ca48afdf233d32bae\": container with ID starting with d4b5ee122224e6d49d9925cc73b552008c719aa2003eaa1ca48afdf233d32bae not found: ID does not exist"
Nov 03 22:23:33.488481 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.488407    5053 scope.go:122] "RemoveContainer" containerID="f5b5c9e2ddfec1c0e0aa0ac52337d096136207eb50c0fbfccfc672afa1e8a28a"
Nov 03 22:23:33.497161 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.497138    5053 scope.go:122] "RemoveContainer" containerID="f5b5c9e2ddfec1c0e0aa0ac52337d096136207eb50c0fbfccfc672afa1e8a28a"
Nov 03 22:23:33.497477 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:23:33.497448    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5b5c9e2ddfec1c0e0aa0ac52337d096136207eb50c0fbfccfc672afa1e8a28a\": container with ID starting with f5b5c9e2ddfec1c0e0aa0ac52337d096136207eb50c0fbfccfc672afa1e8a28a not found: ID does not exist" containerID="f5b5c9e2ddfec1c0e0aa0ac52337d096136207eb50c0fbfccfc672afa1e8a28a"
Nov 03 22:23:33.497542 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.497491    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b5c9e2ddfec1c0e0aa0ac52337d096136207eb50c0fbfccfc672afa1e8a28a"} err="failed to get container status \"f5b5c9e2ddfec1c0e0aa0ac52337d096136207eb50c0fbfccfc672afa1e8a28a\": rpc error: code = NotFound desc = could not find container \"f5b5c9e2ddfec1c0e0aa0ac52337d096136207eb50c0fbfccfc672afa1e8a28a\": container with ID starting with f5b5c9e2ddfec1c0e0aa0ac52337d096136207eb50c0fbfccfc672afa1e8a28a not found: ID does not exist"
Nov 03 22:23:33.591476 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.591423    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt"]
Nov 03 22:23:33.593918 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.593889    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-da5f0b-6d4494c4c6-9dhtt"]
Nov 03 22:23:33.621676 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.621649    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 22:23:33.665462 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.665430    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="770bdb59-84e0-449d-aaea-d75986b816b1" path="/var/lib/kubelet/pods/770bdb59-84e0-449d-aaea-d75986b816b1/volumes"
Nov 03 22:23:33.746904 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.746863    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-data\") pod \"88e0381b-b02d-40f1-acef-babc6c0248d1\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") "
Nov 03 22:23:33.747108 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.746913    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-envoy\") pod \"88e0381b-b02d-40f1-acef-babc6c0248d1\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") "
Nov 03 22:23:33.747108 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.746936    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-workload-certs\") pod \"88e0381b-b02d-40f1-acef-babc6c0248d1\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") "
Nov 03 22:23:33.747108 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.747004    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e0381b-b02d-40f1-acef-babc6c0248d1-kube-api-access-786pf\" (UniqueName: \"kubernetes.io/projected/88e0381b-b02d-40f1-acef-babc6c0248d1-kube-api-access-786pf\") pod \"88e0381b-b02d-40f1-acef-babc6c0248d1\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") "
Nov 03 22:23:33.747108 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.747043    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-token\" (UniqueName: \"kubernetes.io/projected/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-token\") pod \"88e0381b-b02d-40f1-acef-babc6c0248d1\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") "
Nov 03 22:23:33.747108 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.747090    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-credential-socket\") pod \"88e0381b-b02d-40f1-acef-babc6c0248d1\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") "
Nov 03 22:23:33.747428 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.747115    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-podinfo\") pod \"88e0381b-b02d-40f1-acef-babc6c0248d1\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") "
Nov 03 22:23:33.747428 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.747128    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-data" pod "88e0381b-b02d-40f1-acef-babc6c0248d1" (UID: "88e0381b-b02d-40f1-acef-babc6c0248d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:33.747428 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.747159    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e0381b-b02d-40f1-acef-babc6c0248d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e0381b-b02d-40f1-acef-babc6c0248d1-istiod-ca-cert\") pod \"88e0381b-b02d-40f1-acef-babc6c0248d1\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") "
Nov 03 22:23:33.747428 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.747184    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-workload-socket\") pod \"88e0381b-b02d-40f1-acef-babc6c0248d1\" (UID: \"88e0381b-b02d-40f1-acef-babc6c0248d1\") "
Nov 03 22:23:33.747652 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.747555    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-credential-socket" pod "88e0381b-b02d-40f1-acef-babc6c0248d1" (UID: "88e0381b-b02d-40f1-acef-babc6c0248d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:33.747652 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.747618    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-workload-certs" pod "88e0381b-b02d-40f1-acef-babc6c0248d1" (UID: "88e0381b-b02d-40f1-acef-babc6c0248d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:33.747867 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.747848    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e0381b-b02d-40f1-acef-babc6c0248d1-istiod-ca-cert" pod "88e0381b-b02d-40f1-acef-babc6c0248d1" (UID: "88e0381b-b02d-40f1-acef-babc6c0248d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:23:33.747950 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.747853    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:33.747950 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.747893    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:33.747950 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.747909    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:33.748072 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.748059    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-workload-socket" pod "88e0381b-b02d-40f1-acef-babc6c0248d1" (UID: "88e0381b-b02d-40f1-acef-babc6c0248d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:33.748573 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.748548    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h"]
Nov 03 22:23:33.750019 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.749989    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-token" pod "88e0381b-b02d-40f1-acef-babc6c0248d1" (UID: "88e0381b-b02d-40f1-acef-babc6c0248d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:33.750019 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.749991    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-podinfo" pod "88e0381b-b02d-40f1-acef-babc6c0248d1" (UID: "88e0381b-b02d-40f1-acef-babc6c0248d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:23:33.750198 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.750083    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e0381b-b02d-40f1-acef-babc6c0248d1-kube-api-access-786pf" pod "88e0381b-b02d-40f1-acef-babc6c0248d1" (UID: "88e0381b-b02d-40f1-acef-babc6c0248d1"). InnerVolumeSpecName "kube-api-access-786pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:33.751147 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.751129    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-da5f0b-65db6fbbff-7mx6h"]
Nov 03 22:23:33.751274 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.751254    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-envoy" pod "88e0381b-b02d-40f1-acef-babc6c0248d1" (UID: "88e0381b-b02d-40f1-acef-babc6c0248d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:33.848749 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.848716    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:33.848749 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.848743    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e0381b-b02d-40f1-acef-babc6c0248d1-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:33.848749 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.848756    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:33.849005 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.848766    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:33.849005 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.848774    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-786pf\" (UniqueName: \"kubernetes.io/projected/88e0381b-b02d-40f1-acef-babc6c0248d1-kube-api-access-786pf\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:33.849005 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:33.848783    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e0381b-b02d-40f1-acef-babc6c0248d1-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:34.438828 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:34.438795    5053 generic.go:356] "Generic (PLEG): container finished" podID="88e0381b-b02d-40f1-acef-babc6c0248d1" containerID="2b600b493165ff5ed7b542af6da27f090da32cbcbd1c0605b93ccec390622bc8" exitCode=0
Nov 03 22:23:34.439043 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:34.438859    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"
Nov 03 22:23:34.439043 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:34.438866    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj" event={"ID":"88e0381b-b02d-40f1-acef-babc6c0248d1","Type":"ContainerDied","Data":"2b600b493165ff5ed7b542af6da27f090da32cbcbd1c0605b93ccec390622bc8"}
Nov 03 22:23:34.439043 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:34.438923    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj" event={"ID":"88e0381b-b02d-40f1-acef-babc6c0248d1","Type":"ContainerDied","Data":"6731326ae61c08506448aa9202b432f4e5571d28ecb72ba72739097c46d96683"}
Nov 03 22:23:34.439043 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:34.438949    5053 scope.go:122] "RemoveContainer" containerID="2b600b493165ff5ed7b542af6da27f090da32cbcbd1c0605b93ccec390622bc8"
Nov 03 22:23:34.450006 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:34.449984    5053 scope.go:122] "RemoveContainer" containerID="6c9b7abbf29d54e93d7391907f9f88d1229fa2e85431be99a15107ad56df6380"
Nov 03 22:23:34.459043 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:34.459024    5053 scope.go:122] "RemoveContainer" containerID="2b600b493165ff5ed7b542af6da27f090da32cbcbd1c0605b93ccec390622bc8"
Nov 03 22:23:34.459341 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:23:34.459320    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b600b493165ff5ed7b542af6da27f090da32cbcbd1c0605b93ccec390622bc8\": container with ID starting with 2b600b493165ff5ed7b542af6da27f090da32cbcbd1c0605b93ccec390622bc8 not found: ID does not exist" containerID="2b600b493165ff5ed7b542af6da27f090da32cbcbd1c0605b93ccec390622bc8"
Nov 03 22:23:34.459385 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:34.459352    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b600b493165ff5ed7b542af6da27f090da32cbcbd1c0605b93ccec390622bc8"} err="failed to get container status \"2b600b493165ff5ed7b542af6da27f090da32cbcbd1c0605b93ccec390622bc8\": rpc error: code = NotFound desc = could not find container \"2b600b493165ff5ed7b542af6da27f090da32cbcbd1c0605b93ccec390622bc8\": container with ID starting with 2b600b493165ff5ed7b542af6da27f090da32cbcbd1c0605b93ccec390622bc8 not found: ID does not exist"
Nov 03 22:23:34.459385 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:34.459372    5053 scope.go:122] "RemoveContainer" containerID="6c9b7abbf29d54e93d7391907f9f88d1229fa2e85431be99a15107ad56df6380"
Nov 03 22:23:34.459616 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:23:34.459596    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c9b7abbf29d54e93d7391907f9f88d1229fa2e85431be99a15107ad56df6380\": container with ID starting with 6c9b7abbf29d54e93d7391907f9f88d1229fa2e85431be99a15107ad56df6380 not found: ID does not exist" containerID="6c9b7abbf29d54e93d7391907f9f88d1229fa2e85431be99a15107ad56df6380"
Nov 03 22:23:34.459669 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:34.459623    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9b7abbf29d54e93d7391907f9f88d1229fa2e85431be99a15107ad56df6380"} err="failed to get container status \"6c9b7abbf29d54e93d7391907f9f88d1229fa2e85431be99a15107ad56df6380\": rpc error: code = NotFound desc = could not find container \"6c9b7abbf29d54e93d7391907f9f88d1229fa2e85431be99a15107ad56df6380\": container with ID starting with 6c9b7abbf29d54e93d7391907f9f88d1229fa2e85431be99a15107ad56df6380 not found: ID does not exist"
Nov 03 22:23:34.604905 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:34.604862    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"]
Nov 03 22:23:34.653616 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:34.653583    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-868tj"]
Nov 03 22:23:35.067055 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.067023    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-762a773a-wk94p"]
Nov 03 22:23:35.067472 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.067460    5053 cpu_manager.go:414] "containerMap: removing container" podUID="88e0381b-b02d-40f1-acef-babc6c0248d1" containerName="istio-proxy"
Nov 03 22:23:35.067522 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.067474    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e0381b-b02d-40f1-acef-babc6c0248d1" containerName="istio-proxy"
Nov 03 22:23:35.067522 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.067488    5053 cpu_manager.go:414] "containerMap: removing container" podUID="40e7f3d6-f0f4-456e-864e-1c3224bb3463" containerName="payload-pre-processing"
Nov 03 22:23:35.067522 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.067494    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e7f3d6-f0f4-456e-864e-1c3224bb3463" containerName="payload-pre-processing"
Nov 03 22:23:35.067522 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.067513    5053 cpu_manager.go:414] "containerMap: removing container" podUID="770bdb59-84e0-449d-aaea-d75986b816b1" containerName="payload-processing"
Nov 03 22:23:35.067522 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.067517    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="770bdb59-84e0-449d-aaea-d75986b816b1" containerName="payload-processing"
Nov 03 22:23:35.067686 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.067531    5053 cpu_manager.go:414] "containerMap: removing container" podUID="c5f25042-c855-47f7-af24-5c3818afaf80" containerName="revoke-keys"
Nov 03 22:23:35.067686 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.067535    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5f25042-c855-47f7-af24-5c3818afaf80" containerName="revoke-keys"
Nov 03 22:23:35.067686 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.067549    5053 cpu_manager.go:414] "containerMap: removing container" podUID="ce8d8f89-c688-4928-8ea7-ec6b22a2c74e" containerName="payload-processing"
Nov 03 22:23:35.067686 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.067553    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce8d8f89-c688-4928-8ea7-ec6b22a2c74e" containerName="payload-processing"
Nov 03 22:23:35.067686 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.067619    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5f25042-c855-47f7-af24-5c3818afaf80" containerName="revoke-keys"
Nov 03 22:23:35.067686 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.067629    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce8d8f89-c688-4928-8ea7-ec6b22a2c74e" containerName="payload-processing"
Nov 03 22:23:35.067686 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.067636    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e0381b-b02d-40f1-acef-babc6c0248d1" containerName="istio-proxy"
Nov 03 22:23:35.067686 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.067641    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e7f3d6-f0f4-456e-864e-1c3224bb3463" containerName="payload-pre-processing"
Nov 03 22:23:35.067686 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.067649    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e0381b-b02d-40f1-acef-babc6c0248d1" containerName="istio-proxy"
Nov 03 22:23:35.067686 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.067656    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="770bdb59-84e0-449d-aaea-d75986b816b1" containerName="payload-processing"
Nov 03 22:23:35.072525 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.072508    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a773a-wk94p"
Nov 03 22:23:35.122442 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.122410    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cf60cf9-qbdht"]
Nov 03 22:23:35.123123 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.123103    5053 cpu_manager.go:414] "containerMap: removing container" podUID="88e0381b-b02d-40f1-acef-babc6c0248d1" containerName="istio-proxy"
Nov 03 22:23:35.123168 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.123130    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e0381b-b02d-40f1-acef-babc6c0248d1" containerName="istio-proxy"
Nov 03 22:23:35.125126 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.125043    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:23:35.125785 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.125703    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 22:23:35.125908 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.125830    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 22:23:35.126945 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.126919    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf60cf9-qbdht"
Nov 03 22:23:35.128368 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.128344    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-762a773a-wk94p"]
Nov 03 22:23:35.161084 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.161054    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f38e7a8-af78-44fc-bad8-9497fc64fc7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-762a773a-wk94p\" (UID: \"4f38e7a8-af78-44fc-bad8-9497fc64fc7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a773a-wk94p"
Nov 03 22:23:35.161286 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.161102    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7jw9\" (UniqueName: \"kubernetes.io/projected/4f38e7a8-af78-44fc-bad8-9497fc64fc7b-kube-api-access-k7jw9\") pod \"maas-api-revoke-sub-762a773a-wk94p\" (UID: \"4f38e7a8-af78-44fc-bad8-9497fc64fc7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a773a-wk94p"
Nov 03 22:23:35.161286 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.161175    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f38e7a8-af78-44fc-bad8-9497fc64fc7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-762a773a-wk94p\" (UID: \"4f38e7a8-af78-44fc-bad8-9497fc64fc7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a773a-wk94p"
Nov 03 22:23:35.167947 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.167923    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:23:35.180800 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.180775    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cf60cf9-qbdht"]
Nov 03 22:23:35.262280 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.262217    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw95n\" (UniqueName: \"kubernetes.io/projected/4fd82d7c-0aa9-4501-a520-2e92a99b14e4-kube-api-access-rw95n\") pod \"maas-api-revoke-sub-6cf60cf9-qbdht\" (UID: \"4fd82d7c-0aa9-4501-a520-2e92a99b14e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf60cf9-qbdht"
Nov 03 22:23:35.262463 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.262390    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f38e7a8-af78-44fc-bad8-9497fc64fc7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-762a773a-wk94p\" (UID: \"4f38e7a8-af78-44fc-bad8-9497fc64fc7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a773a-wk94p"
Nov 03 22:23:35.262463 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.262447    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd82d7c-0aa9-4501-a520-2e92a99b14e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cf60cf9-qbdht\" (UID: \"4fd82d7c-0aa9-4501-a520-2e92a99b14e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf60cf9-qbdht"
Nov 03 22:23:35.262541 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.262465    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7jw9\" (UniqueName: \"kubernetes.io/projected/4f38e7a8-af78-44fc-bad8-9497fc64fc7b-kube-api-access-k7jw9\") pod \"maas-api-revoke-sub-762a773a-wk94p\" (UID: \"4f38e7a8-af78-44fc-bad8-9497fc64fc7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a773a-wk94p"
Nov 03 22:23:35.262541 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.262487    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f38e7a8-af78-44fc-bad8-9497fc64fc7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-762a773a-wk94p\" (UID: \"4f38e7a8-af78-44fc-bad8-9497fc64fc7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a773a-wk94p"
Nov 03 22:23:35.262541 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.262507    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd82d7c-0aa9-4501-a520-2e92a99b14e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cf60cf9-qbdht\" (UID: \"4fd82d7c-0aa9-4501-a520-2e92a99b14e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf60cf9-qbdht"
Nov 03 22:23:35.263145 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.263129    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f38e7a8-af78-44fc-bad8-9497fc64fc7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-762a773a-wk94p\" (UID: \"4f38e7a8-af78-44fc-bad8-9497fc64fc7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a773a-wk94p"
Nov 03 22:23:35.264926 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.264907    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f38e7a8-af78-44fc-bad8-9497fc64fc7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-762a773a-wk94p\" (UID: \"4f38e7a8-af78-44fc-bad8-9497fc64fc7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a773a-wk94p"
Nov 03 22:23:35.319550 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.319466    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7jw9\" (UniqueName: \"kubernetes.io/projected/4f38e7a8-af78-44fc-bad8-9497fc64fc7b-kube-api-access-k7jw9\") pod \"maas-api-revoke-sub-762a773a-wk94p\" (UID: \"4f38e7a8-af78-44fc-bad8-9497fc64fc7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a773a-wk94p"
Nov 03 22:23:35.363772 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.363735    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd82d7c-0aa9-4501-a520-2e92a99b14e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cf60cf9-qbdht\" (UID: \"4fd82d7c-0aa9-4501-a520-2e92a99b14e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf60cf9-qbdht"
Nov 03 22:23:35.363994 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.363781    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd82d7c-0aa9-4501-a520-2e92a99b14e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cf60cf9-qbdht\" (UID: \"4fd82d7c-0aa9-4501-a520-2e92a99b14e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf60cf9-qbdht"
Nov 03 22:23:35.363994 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.363904    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw95n\" (UniqueName: \"kubernetes.io/projected/4fd82d7c-0aa9-4501-a520-2e92a99b14e4-kube-api-access-rw95n\") pod \"maas-api-revoke-sub-6cf60cf9-qbdht\" (UID: \"4fd82d7c-0aa9-4501-a520-2e92a99b14e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf60cf9-qbdht"
Nov 03 22:23:35.364522 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.364499    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd82d7c-0aa9-4501-a520-2e92a99b14e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cf60cf9-qbdht\" (UID: \"4fd82d7c-0aa9-4501-a520-2e92a99b14e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf60cf9-qbdht"
Nov 03 22:23:35.366285 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.366266    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd82d7c-0aa9-4501-a520-2e92a99b14e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cf60cf9-qbdht\" (UID: \"4fd82d7c-0aa9-4501-a520-2e92a99b14e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf60cf9-qbdht"
Nov 03 22:23:35.385697 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.385671    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a773a-wk94p"
Nov 03 22:23:35.420561 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.420530    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw95n\" (UniqueName: \"kubernetes.io/projected/4fd82d7c-0aa9-4501-a520-2e92a99b14e4-kube-api-access-rw95n\") pod \"maas-api-revoke-sub-6cf60cf9-qbdht\" (UID: \"4fd82d7c-0aa9-4501-a520-2e92a99b14e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf60cf9-qbdht"
Nov 03 22:23:35.443960 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.443927    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf60cf9-qbdht"
Nov 03 22:23:35.545946 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.545905    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd" podUID="82077ea7-1e05-4b6e-806b-a2d2d2a64c4f" containerName="main" containerID="cri-o://55a662149df6a05773b532e8fde81b5a9c9e00ea08f18a093f3581e57e2f513f" gracePeriod=45
Nov 03 22:23:35.665881 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.665841    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40e7f3d6-f0f4-456e-864e-1c3224bb3463" path="/var/lib/kubelet/pods/40e7f3d6-f0f4-456e-864e-1c3224bb3463/volumes"
Nov 03 22:23:35.666425 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.666402    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e0381b-b02d-40f1-acef-babc6c0248d1" path="/var/lib/kubelet/pods/88e0381b-b02d-40f1-acef-babc6c0248d1/volumes"
Nov 03 22:23:35.762039 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.761995    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-762a773a-wk94p"]
Nov 03 22:23:35.764605 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:23:35.764563    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f38e7a8_af78_44fc_bad8_9497fc64fc7b.slice/crio-80b24b845a15819ec1a75cb03bee39fc75dfce20c6e90819e5f03c63ae0f3bae WatchSource:0}: Error finding container 80b24b845a15819ec1a75cb03bee39fc75dfce20c6e90819e5f03c63ae0f3bae: Status 404 returned error can't find the container with id 80b24b845a15819ec1a75cb03bee39fc75dfce20c6e90819e5f03c63ae0f3bae
Nov 03 22:23:35.783768 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.783738    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 22:23:35.861875 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.861852    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cf60cf9-qbdht"]
Nov 03 22:23:35.864050 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:23:35.864027    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd82d7c_0aa9_4501_a520_2e92a99b14e4.slice/crio-ee35f87bbdc1c9438a988e75539bf551078cdc026f03c8fb6a757260fe653a76 WatchSource:0}: Error finding container ee35f87bbdc1c9438a988e75539bf551078cdc026f03c8fb6a757260fe653a76: Status 404 returned error can't find the container with id ee35f87bbdc1c9438a988e75539bf551078cdc026f03c8fb6a757260fe653a76
Nov 03 22:23:35.866963 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.866946    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-home\" (UniqueName: \"kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-home\") pod \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\" (UID: \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\") "
Nov 03 22:23:35.867030 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.866988    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-model-cache\") pod \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\" (UID: \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\") "
Nov 03 22:23:35.867074 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.867027    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-tls-certs\" (UniqueName: \"kubernetes.io/secret/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-tls-certs\") pod \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\" (UID: \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\") "
Nov 03 22:23:35.867187 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.867170    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-kube-api-access-vxk69\" (UniqueName: \"kubernetes.io/projected/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-kube-api-access-vxk69\") pod \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\" (UID: \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\") "
Nov 03 22:23:35.867289 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.867202    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-home" pod "82077ea7-1e05-4b6e-806b-a2d2d2a64c4f" (UID: "82077ea7-1e05-4b6e-806b-a2d2d2a64c4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:35.867366 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.867286    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-tmp-dir\") pod \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\" (UID: \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\") "
Nov 03 22:23:35.867366 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.867294    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-model-cache" pod "82077ea7-1e05-4b6e-806b-a2d2d2a64c4f" (UID: "82077ea7-1e05-4b6e-806b-a2d2d2a64c4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:35.867366 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.867309    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-dshm\") pod \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\" (UID: \"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f\") "
Nov 03 22:23:35.867524 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.867507    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-tmp-dir" pod "82077ea7-1e05-4b6e-806b-a2d2d2a64c4f" (UID: "82077ea7-1e05-4b6e-806b-a2d2d2a64c4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:35.867613 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.867598    5053 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-tmp-dir\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:35.867673 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.867619    5053 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-home\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:35.867673 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.867632    5053 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-model-cache\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:35.870370 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.870346    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-tls-certs" pod "82077ea7-1e05-4b6e-806b-a2d2d2a64c4f" (UID: "82077ea7-1e05-4b6e-806b-a2d2d2a64c4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:23:35.870370 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.870355    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-dshm" pod "82077ea7-1e05-4b6e-806b-a2d2d2a64c4f" (UID: "82077ea7-1e05-4b6e-806b-a2d2d2a64c4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:35.870547 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.870382    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-kube-api-access-vxk69" pod "82077ea7-1e05-4b6e-806b-a2d2d2a64c4f" (UID: "82077ea7-1e05-4b6e-806b-a2d2d2a64c4f"). InnerVolumeSpecName "kube-api-access-vxk69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:35.968629 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.968597    5053 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-tls-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:35.968629 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.968630    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxk69\" (UniqueName: \"kubernetes.io/projected/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-kube-api-access-vxk69\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:35.968872 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:35.968644    5053 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f-dshm\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:36.463473 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:36.463378    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf60cf9-qbdht" event={"ID":"4fd82d7c-0aa9-4501-a520-2e92a99b14e4","Type":"ContainerStarted","Data":"5912f288d827841a1e71849b0b84705e439c1277a2032afd6d228faa0a1fbf5d"}
Nov 03 22:23:36.463473 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:36.463430    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf60cf9-qbdht" event={"ID":"4fd82d7c-0aa9-4501-a520-2e92a99b14e4","Type":"ContainerStarted","Data":"ee35f87bbdc1c9438a988e75539bf551078cdc026f03c8fb6a757260fe653a76"}
Nov 03 22:23:36.464914 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:36.464888    5053 generic.go:356] "Generic (PLEG): container finished" podID="82077ea7-1e05-4b6e-806b-a2d2d2a64c4f" containerID="55a662149df6a05773b532e8fde81b5a9c9e00ea08f18a093f3581e57e2f513f" exitCode=0
Nov 03 22:23:36.465047 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:36.464935    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"
Nov 03 22:23:36.465047 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:36.464961    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd" event={"ID":"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f","Type":"ContainerDied","Data":"55a662149df6a05773b532e8fde81b5a9c9e00ea08f18a093f3581e57e2f513f"}
Nov 03 22:23:36.465047 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:36.464997    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd" event={"ID":"82077ea7-1e05-4b6e-806b-a2d2d2a64c4f","Type":"ContainerDied","Data":"24a7be786893ed61311da1041dd84996fc16c8f5e805b3dcd580d604da3a979d"}
Nov 03 22:23:36.465047 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:36.465013    5053 scope.go:122] "RemoveContainer" containerID="55a662149df6a05773b532e8fde81b5a9c9e00ea08f18a093f3581e57e2f513f"
Nov 03 22:23:36.466778 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:36.466717    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a773a-wk94p" event={"ID":"4f38e7a8-af78-44fc-bad8-9497fc64fc7b","Type":"ContainerStarted","Data":"e69b893c232e0f9cb75a614b7fc32733ef7bf888b535e942dfb80cd6e9c0b105"}
Nov 03 22:23:36.466778 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:36.466768    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a773a-wk94p" event={"ID":"4f38e7a8-af78-44fc-bad8-9497fc64fc7b","Type":"ContainerStarted","Data":"80b24b845a15819ec1a75cb03bee39fc75dfce20c6e90819e5f03c63ae0f3bae"}
Nov 03 22:23:36.481474 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:36.479272    5053 scope.go:122] "RemoveContainer" containerID="55a662149df6a05773b532e8fde81b5a9c9e00ea08f18a093f3581e57e2f513f"
Nov 03 22:23:36.481474 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:23:36.479674    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a662149df6a05773b532e8fde81b5a9c9e00ea08f18a093f3581e57e2f513f\": container with ID starting with 55a662149df6a05773b532e8fde81b5a9c9e00ea08f18a093f3581e57e2f513f not found: ID does not exist" containerID="55a662149df6a05773b532e8fde81b5a9c9e00ea08f18a093f3581e57e2f513f"
Nov 03 22:23:36.481474 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:36.479720    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a662149df6a05773b532e8fde81b5a9c9e00ea08f18a093f3581e57e2f513f"} err="failed to get container status \"55a662149df6a05773b532e8fde81b5a9c9e00ea08f18a093f3581e57e2f513f\": rpc error: code = NotFound desc = could not find container \"55a662149df6a05773b532e8fde81b5a9c9e00ea08f18a093f3581e57e2f513f\": container with ID starting with 55a662149df6a05773b532e8fde81b5a9c9e00ea08f18a093f3581e57e2f513f not found: ID does not exist"
Nov 03 22:23:36.579797 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:36.579747    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf60cf9-qbdht" podStartSLOduration=1.579733536 podStartE2EDuration="1.579733536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:36.577592396 +0000 UTC m=+4131.071514288" watchObservedRunningTime="2026-11-03 22:23:36.579733536 +0000 UTC m=+4131.073655411"
Nov 03 22:23:36.688621 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:36.688587    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"]
Nov 03 22:23:36.697292 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:36.697231    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-da5f0b/distinct-w3-da5f0b-kserve-8647556fc6-zsrmd"]
Nov 03 22:23:36.810186 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:36.810131    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a773a-wk94p" podStartSLOduration=1.810117239 podStartE2EDuration="1.810117239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:36.807174548 +0000 UTC m=+4131.301096423" watchObservedRunningTime="2026-11-03 22:23:36.810117239 +0000 UTC m=+4131.304039113"
Nov 03 22:23:37.474230 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:37.474194    5053 generic.go:356] "Generic (PLEG): container finished" podID="4fd82d7c-0aa9-4501-a520-2e92a99b14e4" containerID="5912f288d827841a1e71849b0b84705e439c1277a2032afd6d228faa0a1fbf5d" exitCode=0
Nov 03 22:23:37.474469 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:37.474274    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf60cf9-qbdht" event={"ID":"4fd82d7c-0aa9-4501-a520-2e92a99b14e4","Type":"ContainerDied","Data":"5912f288d827841a1e71849b0b84705e439c1277a2032afd6d228faa0a1fbf5d"}
Nov 03 22:23:37.476644 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:37.476618    5053 generic.go:356] "Generic (PLEG): container finished" podID="4f38e7a8-af78-44fc-bad8-9497fc64fc7b" containerID="e69b893c232e0f9cb75a614b7fc32733ef7bf888b535e942dfb80cd6e9c0b105" exitCode=0
Nov 03 22:23:37.476768 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:37.476697    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a773a-wk94p" event={"ID":"4f38e7a8-af78-44fc-bad8-9497fc64fc7b","Type":"ContainerDied","Data":"e69b893c232e0f9cb75a614b7fc32733ef7bf888b535e942dfb80cd6e9c0b105"}
Nov 03 22:23:37.666147 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:37.666117    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82077ea7-1e05-4b6e-806b-a2d2d2a64c4f" path="/var/lib/kubelet/pods/82077ea7-1e05-4b6e-806b-a2d2d2a64c4f/volumes"
Nov 03 22:23:38.672300 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:38.672270    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a773a-wk94p"
Nov 03 22:23:38.674858 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:38.674836    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf60cf9-qbdht"
Nov 03 22:23:38.802409 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:38.802280    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fd82d7c-0aa9-4501-a520-2e92a99b14e4-kube-api-access-rw95n\" (UniqueName: \"kubernetes.io/projected/4fd82d7c-0aa9-4501-a520-2e92a99b14e4-kube-api-access-rw95n\") pod \"4fd82d7c-0aa9-4501-a520-2e92a99b14e4\" (UID: \"4fd82d7c-0aa9-4501-a520-2e92a99b14e4\") "
Nov 03 22:23:38.802409 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:38.802333    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f38e7a8-af78-44fc-bad8-9497fc64fc7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f38e7a8-af78-44fc-bad8-9497fc64fc7b-maas-api-xks-ca\") pod \"4f38e7a8-af78-44fc-bad8-9497fc64fc7b\" (UID: \"4f38e7a8-af78-44fc-bad8-9497fc64fc7b\") "
Nov 03 22:23:38.802409 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:38.802358    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f38e7a8-af78-44fc-bad8-9497fc64fc7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f38e7a8-af78-44fc-bad8-9497fc64fc7b-maas-api-openshift-ca\") pod \"4f38e7a8-af78-44fc-bad8-9497fc64fc7b\" (UID: \"4f38e7a8-af78-44fc-bad8-9497fc64fc7b\") "
Nov 03 22:23:38.802409 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:38.802394    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fd82d7c-0aa9-4501-a520-2e92a99b14e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd82d7c-0aa9-4501-a520-2e92a99b14e4-maas-api-xks-ca\") pod \"4fd82d7c-0aa9-4501-a520-2e92a99b14e4\" (UID: \"4fd82d7c-0aa9-4501-a520-2e92a99b14e4\") "
Nov 03 22:23:38.803039 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:38.802423    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fd82d7c-0aa9-4501-a520-2e92a99b14e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd82d7c-0aa9-4501-a520-2e92a99b14e4-maas-api-openshift-ca\") pod \"4fd82d7c-0aa9-4501-a520-2e92a99b14e4\" (UID: \"4fd82d7c-0aa9-4501-a520-2e92a99b14e4\") "
Nov 03 22:23:38.803039 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:38.802511    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f38e7a8-af78-44fc-bad8-9497fc64fc7b-kube-api-access-k7jw9\" (UniqueName: \"kubernetes.io/projected/4f38e7a8-af78-44fc-bad8-9497fc64fc7b-kube-api-access-k7jw9\") pod \"4f38e7a8-af78-44fc-bad8-9497fc64fc7b\" (UID: \"4f38e7a8-af78-44fc-bad8-9497fc64fc7b\") "
Nov 03 22:23:38.803039 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:38.802898    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f38e7a8-af78-44fc-bad8-9497fc64fc7b-maas-api-openshift-ca" pod "4f38e7a8-af78-44fc-bad8-9497fc64fc7b" (UID: "4f38e7a8-af78-44fc-bad8-9497fc64fc7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:23:38.803284 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:38.803103    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f38e7a8-af78-44fc-bad8-9497fc64fc7b-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:38.804095 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:38.804062    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd82d7c-0aa9-4501-a520-2e92a99b14e4-maas-api-openshift-ca" pod "4fd82d7c-0aa9-4501-a520-2e92a99b14e4" (UID: "4fd82d7c-0aa9-4501-a520-2e92a99b14e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:23:38.806622 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:38.806262    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f38e7a8-af78-44fc-bad8-9497fc64fc7b-maas-api-xks-ca" pod "4f38e7a8-af78-44fc-bad8-9497fc64fc7b" (UID: "4f38e7a8-af78-44fc-bad8-9497fc64fc7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:23:38.806622 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:38.806606    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f38e7a8-af78-44fc-bad8-9497fc64fc7b-kube-api-access-k7jw9" pod "4f38e7a8-af78-44fc-bad8-9497fc64fc7b" (UID: "4f38e7a8-af78-44fc-bad8-9497fc64fc7b"). InnerVolumeSpecName "kube-api-access-k7jw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:38.806823 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:38.806634    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd82d7c-0aa9-4501-a520-2e92a99b14e4-kube-api-access-rw95n" pod "4fd82d7c-0aa9-4501-a520-2e92a99b14e4" (UID: "4fd82d7c-0aa9-4501-a520-2e92a99b14e4"). InnerVolumeSpecName "kube-api-access-rw95n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:38.806823 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:38.806715    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd82d7c-0aa9-4501-a520-2e92a99b14e4-maas-api-xks-ca" pod "4fd82d7c-0aa9-4501-a520-2e92a99b14e4" (UID: "4fd82d7c-0aa9-4501-a520-2e92a99b14e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:23:38.904638 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:38.904597    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw95n\" (UniqueName: \"kubernetes.io/projected/4fd82d7c-0aa9-4501-a520-2e92a99b14e4-kube-api-access-rw95n\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:38.904638 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:38.904642    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f38e7a8-af78-44fc-bad8-9497fc64fc7b-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:38.905052 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:38.904658    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd82d7c-0aa9-4501-a520-2e92a99b14e4-maas-api-xks-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:38.905052 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:38.904672    5053 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd82d7c-0aa9-4501-a520-2e92a99b14e4-maas-api-openshift-ca\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:38.905052 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:38.904686    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7jw9\" (UniqueName: \"kubernetes.io/projected/4f38e7a8-af78-44fc-bad8-9497fc64fc7b-kube-api-access-k7jw9\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:39.488299 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:39.488273    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf60cf9-qbdht"
Nov 03 22:23:39.488502 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:39.488297    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf60cf9-qbdht" event={"ID":"4fd82d7c-0aa9-4501-a520-2e92a99b14e4","Type":"ContainerDied","Data":"ee35f87bbdc1c9438a988e75539bf551078cdc026f03c8fb6a757260fe653a76"}
Nov 03 22:23:39.488502 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:39.488346    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee35f87bbdc1c9438a988e75539bf551078cdc026f03c8fb6a757260fe653a76"
Nov 03 22:23:39.489759 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:39.489739    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a773a-wk94p"
Nov 03 22:23:39.489880 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:39.489771    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762a773a-wk94p" event={"ID":"4f38e7a8-af78-44fc-bad8-9497fc64fc7b","Type":"ContainerDied","Data":"80b24b845a15819ec1a75cb03bee39fc75dfce20c6e90819e5f03c63ae0f3bae"}
Nov 03 22:23:39.489880 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:39.489808    5053 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80b24b845a15819ec1a75cb03bee39fc75dfce20c6e90819e5f03c63ae0f3bae"
Nov 03 22:23:40.927681 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:23:40.927643    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:23:48.477729 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:48.477701    5053 scope.go:122] "RemoveContainer" containerID="e93a792cd716e63ad9e231e80debee828f7d8e219328b48c3daed07f13e63bf7"
Nov 03 22:23:48.487316 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:48.487294    5053 scope.go:122] "RemoveContainer" containerID="cd8dc383b439f79e13e223c48e54410069c13ff033541735d99b9980018e9fda"
Nov 03 22:23:52.465903 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:52.465862    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj"]
Nov 03 22:23:52.515043 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:52.515007    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-da5f0b-5s6zj"]
Nov 03 22:23:53.668861 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:53.668824    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5f25042-c855-47f7-af24-5c3818afaf80" path="/var/lib/kubelet/pods/c5f25042-c855-47f7-af24-5c3818afaf80/volumes"
Nov 03 22:23:59.963705 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:59.963670    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"]
Nov 03 22:23:59.964196 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:23:59.963866    5053 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv" podUID="0a1fdc06-686e-499e-9c23-ae901773d3b6" containerName="istio-proxy" containerID="cri-o://89da477e9c050923dbf5bcce3b3a9e3e812b78311979ac0a38bd896a6ad4e15b" gracePeriod=30
Nov 03 22:24:01.494685 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:01.494654    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-jnjhr_bcdcbe12-1bff-4947-ab42-909bd405e7b1/manager/0.log"
Nov 03 22:24:01.760883 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:01.760802    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-7twr6_79d0e65f-9499-4fd9-8b3c-6f70bd3468f1/manager/0.log"
Nov 03 22:24:02.926896 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:02.926862    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-ngwtq_9e988f30-c8ac-4357-9d38-a7720a906f06/manager/0.log"
Nov 03 22:24:03.549219 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:03.549186    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-qrdkr_757361e7-9919-408f-9377-317923c5ffa5/manager/0.log"
Nov 03 22:24:04.350609 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:04.350578    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-96cd8748-ck6m2_a1461a40-86ef-4281-bb1f-b157e7b4e2f3/manager/0.log"
Nov 03 22:24:04.589147 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:04.589117    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-vmp59_36ec7139-4665-4a3c-ab8e-5ac33799ca58/manager/0.log"
Nov 03 22:24:04.821185 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:04.821152    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-kmgzq_1028ce52-b2d5-4d38-a449-b3396a58d4db/manager/0.log"
Nov 03 22:24:05.219172 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.219149    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 22:24:05.334112 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.334022    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-envoy\") pod \"0a1fdc06-686e-499e-9c23-ae901773d3b6\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") "
Nov 03 22:24:05.334112 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.334061    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-data\") pod \"0a1fdc06-686e-499e-9c23-ae901773d3b6\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") "
Nov 03 22:24:05.334112 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.334081    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-credential-socket\") pod \"0a1fdc06-686e-499e-9c23-ae901773d3b6\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") "
Nov 03 22:24:05.334447 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.334120    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-workload-certs\") pod \"0a1fdc06-686e-499e-9c23-ae901773d3b6\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") "
Nov 03 22:24:05.334447 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.334140    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a1fdc06-686e-499e-9c23-ae901773d3b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1fdc06-686e-499e-9c23-ae901773d3b6-istiod-ca-cert\") pod \"0a1fdc06-686e-499e-9c23-ae901773d3b6\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") "
Nov 03 22:24:05.334447 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.334187    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a1fdc06-686e-499e-9c23-ae901773d3b6-kube-api-access-n5znl\" (UniqueName: \"kubernetes.io/projected/0a1fdc06-686e-499e-9c23-ae901773d3b6-kube-api-access-n5znl\") pod \"0a1fdc06-686e-499e-9c23-ae901773d3b6\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") "
Nov 03 22:24:05.334447 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.334218    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-podinfo\") pod \"0a1fdc06-686e-499e-9c23-ae901773d3b6\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") "
Nov 03 22:24:05.334447 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.334384    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-data" pod "0a1fdc06-686e-499e-9c23-ae901773d3b6" (UID: "0a1fdc06-686e-499e-9c23-ae901773d3b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:24:05.334715 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.334459    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-token\" (UniqueName: \"kubernetes.io/projected/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-token\") pod \"0a1fdc06-686e-499e-9c23-ae901773d3b6\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") "
Nov 03 22:24:05.334715 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.334486    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-workload-certs" pod "0a1fdc06-686e-499e-9c23-ae901773d3b6" (UID: "0a1fdc06-686e-499e-9c23-ae901773d3b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:24:05.334715 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.334496    5053 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-workload-socket\") pod \"0a1fdc06-686e-499e-9c23-ae901773d3b6\" (UID: \"0a1fdc06-686e-499e-9c23-ae901773d3b6\") "
Nov 03 22:24:05.334715 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.334510    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-credential-socket" pod "0a1fdc06-686e-499e-9c23-ae901773d3b6" (UID: "0a1fdc06-686e-499e-9c23-ae901773d3b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:24:05.334715 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.334591    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a1fdc06-686e-499e-9c23-ae901773d3b6-istiod-ca-cert" pod "0a1fdc06-686e-499e-9c23-ae901773d3b6" (UID: "0a1fdc06-686e-499e-9c23-ae901773d3b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:24:05.335149 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.334807    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-workload-socket" pod "0a1fdc06-686e-499e-9c23-ae901773d3b6" (UID: "0a1fdc06-686e-499e-9c23-ae901773d3b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:24:05.335149 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.334895    5053 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-workload-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:24:05.335149 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.334913    5053 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-data\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:24:05.335149 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.334922    5053 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-credential-socket\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:24:05.335149 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.334930    5053 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-workload-certs\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:24:05.335149 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.334940    5053 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1fdc06-686e-499e-9c23-ae901773d3b6-istiod-ca-cert\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:24:05.336497 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.336465    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-envoy" pod "0a1fdc06-686e-499e-9c23-ae901773d3b6" (UID: "0a1fdc06-686e-499e-9c23-ae901773d3b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:24:05.336605 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.336570    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1fdc06-686e-499e-9c23-ae901773d3b6-kube-api-access-n5znl" pod "0a1fdc06-686e-499e-9c23-ae901773d3b6" (UID: "0a1fdc06-686e-499e-9c23-ae901773d3b6"). InnerVolumeSpecName "kube-api-access-n5znl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:24:05.336958 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.336943    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-token" pod "0a1fdc06-686e-499e-9c23-ae901773d3b6" (UID: "0a1fdc06-686e-499e-9c23-ae901773d3b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:24:05.337028 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.336966    5053 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-podinfo" pod "0a1fdc06-686e-499e-9c23-ae901773d3b6" (UID: "0a1fdc06-686e-499e-9c23-ae901773d3b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:24:05.435647 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.435613    5053 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-envoy\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:24:05.435647 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.435644    5053 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5znl\" (UniqueName: \"kubernetes.io/projected/0a1fdc06-686e-499e-9c23-ae901773d3b6-kube-api-access-n5znl\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:24:05.436065 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.435659    5053 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-podinfo\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:24:05.436065 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.435672    5053 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1fdc06-686e-499e-9c23-ae901773d3b6-istio-token\") on node \"ip-10-0-128-43.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:24:05.644716 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.644618    5053 generic.go:356] "Generic (PLEG): container finished" podID="0a1fdc06-686e-499e-9c23-ae901773d3b6" containerID="89da477e9c050923dbf5bcce3b3a9e3e812b78311979ac0a38bd896a6ad4e15b" exitCode=0
Nov 03 22:24:05.644716 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.644673    5053 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"
Nov 03 22:24:05.644910 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.644703    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv" event={"ID":"0a1fdc06-686e-499e-9c23-ae901773d3b6","Type":"ContainerDied","Data":"89da477e9c050923dbf5bcce3b3a9e3e812b78311979ac0a38bd896a6ad4e15b"}
Nov 03 22:24:05.644910 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.644770    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv" event={"ID":"0a1fdc06-686e-499e-9c23-ae901773d3b6","Type":"ContainerDied","Data":"2a81d1e779df173196eeb68fbc09984220e4f3e50a1899a86659144007c04930"}
Nov 03 22:24:05.644910 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.644798    5053 scope.go:122] "RemoveContainer" containerID="89da477e9c050923dbf5bcce3b3a9e3e812b78311979ac0a38bd896a6ad4e15b"
Nov 03 22:24:05.655204 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.655182    5053 scope.go:122] "RemoveContainer" containerID="83a33949ac671af1b3a7c54eaac732d1b65b39f42b70dd1145e73e94809fdd63"
Nov 03 22:24:05.665649 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.665630    5053 scope.go:122] "RemoveContainer" containerID="89da477e9c050923dbf5bcce3b3a9e3e812b78311979ac0a38bd896a6ad4e15b"
Nov 03 22:24:05.665928 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:24:05.665909    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89da477e9c050923dbf5bcce3b3a9e3e812b78311979ac0a38bd896a6ad4e15b\": container with ID starting with 89da477e9c050923dbf5bcce3b3a9e3e812b78311979ac0a38bd896a6ad4e15b not found: ID does not exist" containerID="89da477e9c050923dbf5bcce3b3a9e3e812b78311979ac0a38bd896a6ad4e15b"
Nov 03 22:24:05.666002 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.665937    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89da477e9c050923dbf5bcce3b3a9e3e812b78311979ac0a38bd896a6ad4e15b"} err="failed to get container status \"89da477e9c050923dbf5bcce3b3a9e3e812b78311979ac0a38bd896a6ad4e15b\": rpc error: code = NotFound desc = could not find container \"89da477e9c050923dbf5bcce3b3a9e3e812b78311979ac0a38bd896a6ad4e15b\": container with ID starting with 89da477e9c050923dbf5bcce3b3a9e3e812b78311979ac0a38bd896a6ad4e15b not found: ID does not exist"
Nov 03 22:24:05.666002 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.665957    5053 scope.go:122] "RemoveContainer" containerID="83a33949ac671af1b3a7c54eaac732d1b65b39f42b70dd1145e73e94809fdd63"
Nov 03 22:24:05.666214 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:24:05.666195    5053 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83a33949ac671af1b3a7c54eaac732d1b65b39f42b70dd1145e73e94809fdd63\": container with ID starting with 83a33949ac671af1b3a7c54eaac732d1b65b39f42b70dd1145e73e94809fdd63 not found: ID does not exist" containerID="83a33949ac671af1b3a7c54eaac732d1b65b39f42b70dd1145e73e94809fdd63"
Nov 03 22:24:05.666314 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.666218    5053 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a33949ac671af1b3a7c54eaac732d1b65b39f42b70dd1145e73e94809fdd63"} err="failed to get container status \"83a33949ac671af1b3a7c54eaac732d1b65b39f42b70dd1145e73e94809fdd63\": rpc error: code = NotFound desc = could not find container \"83a33949ac671af1b3a7c54eaac732d1b65b39f42b70dd1145e73e94809fdd63\": container with ID starting with 83a33949ac671af1b3a7c54eaac732d1b65b39f42b70dd1145e73e94809fdd63 not found: ID does not exist"
Nov 03 22:24:05.808959 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.808916    5053 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"]
Nov 03 22:24:05.859038 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:05.859003    5053 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-8bksv"]
Nov 03 22:24:06.331928 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:06.331894    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-z5jk4_a3134aab-ba99-4e98-974a-dc26c5966860/server/0.log"
Nov 03 22:24:06.564159 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:06.564127    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-h2zs5_53dadcdb-7f9d-4dd1-acb8-9321d2bf3ef3/manager/0.log"
Nov 03 22:24:06.821498 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:06.821468    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-pt78x_01bb8d54-6cba-4a24-ac55-4e15a54b87db/proxy/0.log"
Nov 03 22:24:07.664922 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:07.664887    5053 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a1fdc06-686e-499e-9c23-ae901773d3b6" path="/var/lib/kubelet/pods/0a1fdc06-686e-499e-9c23-ae901773d3b6/volumes"
Nov 03 22:24:10.929672 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:24:10.929635    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:24:18.245289 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:18.245258    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-96cd8748-ck6m2_a1461a40-86ef-4281-bb1f-b157e7b4e2f3/manager/0.log"
Nov 03 22:24:18.612590 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:18.612511    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-vmp59_36ec7139-4665-4a3c-ab8e-5ac33799ca58/manager/0.log"
Nov 03 22:24:18.854981 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:18.854950    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-kmgzq_1028ce52-b2d5-4d38-a449-b3396a58d4db/manager/0.log"
Nov 03 22:24:20.069271 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:20.069226    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-z5jk4_a3134aab-ba99-4e98-974a-dc26c5966860/server/0.log"
Nov 03 22:24:20.293036 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:20.293007    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-h2zs5_53dadcdb-7f9d-4dd1-acb8-9321d2bf3ef3/manager/0.log"
Nov 03 22:24:20.665126 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:20.665094    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-pt78x_01bb8d54-6cba-4a24-ac55-4e15a54b87db/proxy/0.log"
Nov 03 22:24:21.757541 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:21.757515    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0f9432e8-q62fh_a65b845d-c808-431e-b18d-976a3f93604f/revoke-keys/0.log"
Nov 03 22:24:21.986643 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:21.986608    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-27aa713a-7dbw4_b43bd20d-e094-41be-8491-779f15085eb5/revoke-keys/0.log"
Nov 03 22:24:22.203757 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:22.203735    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2f514a6e-pc52f_6ae36d0a-e7ff-4f6b-84b0-999559282e87/revoke-keys/0.log"
Nov 03 22:24:22.656693 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:22.656611    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-59888f74-zpnhp_3a117058-0ed7-496b-bc07-68dd892f319e/revoke-keys/0.log"
Nov 03 22:24:22.871161 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:22.871120    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6cf60cf9-qbdht_4fd82d7c-0aa9-4501-a520-2e92a99b14e4/revoke-keys/0.log"
Nov 03 22:24:23.089702 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:23.089671    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6d6fe726-4j2sv_90d53cf2-e5d8-4561-b87a-bcbd9485f141/revoke-keys/0.log"
Nov 03 22:24:23.324511 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:23.324477    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-762a773a-wk94p_4f38e7a8-af78-44fc-bad8-9497fc64fc7b/revoke-keys/0.log"
Nov 03 22:24:25.972287 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:25.972189    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-jnjhr_bcdcbe12-1bff-4947-ab42-909bd405e7b1/manager/0.log"
Nov 03 22:24:26.299641 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:26.299561    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-7twr6_79d0e65f-9499-4fd9-8b3c-6f70bd3468f1/manager/0.log"
Nov 03 22:24:27.330151 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:27.330121    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-ngwtq_9e988f30-c8ac-4357-9d38-a7720a906f06/manager/0.log"
Nov 03 22:24:27.784061 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:27.784031    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-qrdkr_757361e7-9919-408f-9377-317923c5ffa5/manager/0.log"
Nov 03 22:24:28.594387 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:28.594353    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-58c98c4485fcxmh_e8886b8b-7d2c-4b60-8843-651e85091077/istio-proxy/0.log"
Nov 03 22:24:30.153958 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:30.153923    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9_5665aa62-30d5-4ced-8b5c-fedb882823cf/istio-proxy/0.log"
Nov 03 22:24:30.802314 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:30.802278    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5ffbbbfcf7-99qbj_7ccca940-de12-451b-b920-d6bef516942f/router/0.log"
Nov 03 22:24:31.952046 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:31.952014    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-x8csc_5a2bbba0-9cc7-4f82-9803-abfc59037289/main/0.log"
Nov 03 22:24:32.444895 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:32.444868    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-btvm4_1aad4a0a-f17b-48bf-91bd-7f3e193e83a8/main/0.log"
Nov 03 22:24:32.668939 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:32.668909    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w2g4h_1b4c0b5f-ee49-4cb5-bc1e-fe80052bb1e2/main/0.log"
Nov 03 22:24:32.802231 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:32.802151    5053 generic.go:356] "Generic (PLEG): container finished" podID="9e988f30-c8ac-4357-9d38-a7720a906f06" containerID="2fcd78892e27c7de27fd7a800c7a7acbc66da158697446c8940f4970057cd858" exitCode=0
Nov 03 22:24:32.802415 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:32.802221    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngwtq" event={"ID":"9e988f30-c8ac-4357-9d38-a7720a906f06","Type":"ContainerDied","Data":"2fcd78892e27c7de27fd7a800c7a7acbc66da158697446c8940f4970057cd858"}
Nov 03 22:24:32.802607 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:32.802593    5053 scope.go:122] "RemoveContainer" containerID="2fcd78892e27c7de27fd7a800c7a7acbc66da158697446c8940f4970057cd858"
Nov 03 22:24:33.158560 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:33.158485    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-pnh75_24f5f506-2f6d-40f9-a692-5e62af63d3d1/main/0.log"
Nov 03 22:24:33.809652 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:33.809616    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngwtq" event={"ID":"9e988f30-c8ac-4357-9d38-a7720a906f06","Type":"ContainerStarted","Data":"120f55323c5e5667c59b4a7c2d647e3ee9a37dba162d9cabcd84028e6ca6ae46"}
Nov 03 22:24:33.809840 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:33.809828    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngwtq"
Nov 03 22:24:40.926677 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:24:40.926630    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:24:43.867318 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.867287    5053 generic.go:356] "Generic (PLEG): container finished" podID="9e988f30-c8ac-4357-9d38-a7720a906f06" containerID="120f55323c5e5667c59b4a7c2d647e3ee9a37dba162d9cabcd84028e6ca6ae46" exitCode=1
Nov 03 22:24:43.867840 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.867355    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngwtq" event={"ID":"9e988f30-c8ac-4357-9d38-a7720a906f06","Type":"ContainerDied","Data":"120f55323c5e5667c59b4a7c2d647e3ee9a37dba162d9cabcd84028e6ca6ae46"}
Nov 03 22:24:43.867840 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.867407    5053 scope.go:122] "RemoveContainer" containerID="2fcd78892e27c7de27fd7a800c7a7acbc66da158697446c8940f4970057cd858"
Nov 03 22:24:43.867978 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.867951    5053 scope.go:122] "RemoveContainer" containerID="120f55323c5e5667c59b4a7c2d647e3ee9a37dba162d9cabcd84028e6ca6ae46"
Nov 03 22:24:43.868335 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:24:43.868308    5053 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kuadrant-operator-controller-manager-678d87766f-ngwtq_kuadrant-system(9e988f30-c8ac-4357-9d38-a7720a906f06)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngwtq" podUID="9e988f30-c8ac-4357-9d38-a7720a906f06"
Nov 03 22:24:43.877185 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.877159    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gs8dn/must-gather-jq762"]
Nov 03 22:24:43.877787 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.877767    5053 cpu_manager.go:414] "containerMap: removing container" podUID="0a1fdc06-686e-499e-9c23-ae901773d3b6" containerName="istio-proxy"
Nov 03 22:24:43.877787 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.877786    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a1fdc06-686e-499e-9c23-ae901773d3b6" containerName="istio-proxy"
Nov 03 22:24:43.877952 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.877804    5053 cpu_manager.go:414] "containerMap: removing container" podUID="4fd82d7c-0aa9-4501-a520-2e92a99b14e4" containerName="revoke-keys"
Nov 03 22:24:43.877952 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.877809    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd82d7c-0aa9-4501-a520-2e92a99b14e4" containerName="revoke-keys"
Nov 03 22:24:43.877952 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.877829    5053 cpu_manager.go:414] "containerMap: removing container" podUID="4f38e7a8-af78-44fc-bad8-9497fc64fc7b" containerName="revoke-keys"
Nov 03 22:24:43.877952 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.877835    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f38e7a8-af78-44fc-bad8-9497fc64fc7b" containerName="revoke-keys"
Nov 03 22:24:43.877952 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.877843    5053 cpu_manager.go:414] "containerMap: removing container" podUID="82077ea7-1e05-4b6e-806b-a2d2d2a64c4f" containerName="main"
Nov 03 22:24:43.877952 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.877848    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82077ea7-1e05-4b6e-806b-a2d2d2a64c4f" containerName="main"
Nov 03 22:24:43.877952 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.877854    5053 cpu_manager.go:414] "containerMap: removing container" podUID="0a1fdc06-686e-499e-9c23-ae901773d3b6" containerName="istio-proxy"
Nov 03 22:24:43.877952 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.877858    5053 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a1fdc06-686e-499e-9c23-ae901773d3b6" containerName="istio-proxy"
Nov 03 22:24:43.877952 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.877948    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a1fdc06-686e-499e-9c23-ae901773d3b6" containerName="istio-proxy"
Nov 03 22:24:43.878544 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.877959    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd82d7c-0aa9-4501-a520-2e92a99b14e4" containerName="revoke-keys"
Nov 03 22:24:43.878544 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.877966    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f38e7a8-af78-44fc-bad8-9497fc64fc7b" containerName="revoke-keys"
Nov 03 22:24:43.878544 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.877974    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="82077ea7-1e05-4b6e-806b-a2d2d2a64c4f" containerName="main"
Nov 03 22:24:43.878544 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.877984    5053 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a1fdc06-686e-499e-9c23-ae901773d3b6" containerName="istio-proxy"
Nov 03 22:24:43.881385 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.881367    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gs8dn/must-gather-jq762"
Nov 03 22:24:43.896528 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.896502    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wbgf\" (UniqueName: \"kubernetes.io/projected/5fb5b8ff-178f-4c9a-951a-dd228d57ef4b-kube-api-access-4wbgf\") pod \"must-gather-jq762\" (UID: \"5fb5b8ff-178f-4c9a-951a-dd228d57ef4b\") " pod="openshift-must-gather-gs8dn/must-gather-jq762"
Nov 03 22:24:43.896652 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.896616    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-178f-4c9a-951a-dd228d57ef4b-must-gather-output\") pod \"must-gather-jq762\" (UID: \"5fb5b8ff-178f-4c9a-951a-dd228d57ef4b\") " pod="openshift-must-gather-gs8dn/must-gather-jq762"
Nov 03 22:24:43.969355 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.969311    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gs8dn\"/\"openshift-service-ca.crt\""
Nov 03 22:24:43.969544 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.969420    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gs8dn\"/\"default-dockercfg-4c6bh\""
Nov 03 22:24:43.971579 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.971560    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gs8dn/must-gather-jq762"]
Nov 03 22:24:43.973860 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.973839    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gs8dn\"/\"kube-root-ca.crt\""
Nov 03 22:24:43.997854 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.997819    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wbgf\" (UniqueName: \"kubernetes.io/projected/5fb5b8ff-178f-4c9a-951a-dd228d57ef4b-kube-api-access-4wbgf\") pod \"must-gather-jq762\" (UID: \"5fb5b8ff-178f-4c9a-951a-dd228d57ef4b\") " pod="openshift-must-gather-gs8dn/must-gather-jq762"
Nov 03 22:24:43.998051 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.997889    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-178f-4c9a-951a-dd228d57ef4b-must-gather-output\") pod \"must-gather-jq762\" (UID: \"5fb5b8ff-178f-4c9a-951a-dd228d57ef4b\") " pod="openshift-must-gather-gs8dn/must-gather-jq762"
Nov 03 22:24:43.998278 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:43.998223    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-178f-4c9a-951a-dd228d57ef4b-must-gather-output\") pod \"must-gather-jq762\" (UID: \"5fb5b8ff-178f-4c9a-951a-dd228d57ef4b\") " pod="openshift-must-gather-gs8dn/must-gather-jq762"
Nov 03 22:24:44.141761 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:44.141684    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wbgf\" (UniqueName: \"kubernetes.io/projected/5fb5b8ff-178f-4c9a-951a-dd228d57ef4b-kube-api-access-4wbgf\") pod \"must-gather-jq762\" (UID: \"5fb5b8ff-178f-4c9a-951a-dd228d57ef4b\") " pod="openshift-must-gather-gs8dn/must-gather-jq762"
Nov 03 22:24:44.205678 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:44.205646    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gs8dn/must-gather-jq762"
Nov 03 22:24:44.575052 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:44.575022    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gs8dn/must-gather-jq762"]
Nov 03 22:24:44.576901 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:24:44.576874    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fb5b8ff_178f_4c9a_951a_dd228d57ef4b.slice/crio-0ee77e7fbddd4494ff09533e566878913d5c271f415f695eca24dab2b9bc5bd8 WatchSource:0}: Error finding container 0ee77e7fbddd4494ff09533e566878913d5c271f415f695eca24dab2b9bc5bd8: Status 404 returned error can't find the container with id 0ee77e7fbddd4494ff09533e566878913d5c271f415f695eca24dab2b9bc5bd8
Nov 03 22:24:44.578702 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:44.578686    5053 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 22:24:44.876348 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:44.876265    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gs8dn/must-gather-jq762" event={"ID":"5fb5b8ff-178f-4c9a-951a-dd228d57ef4b","Type":"ContainerStarted","Data":"0ee77e7fbddd4494ff09533e566878913d5c271f415f695eca24dab2b9bc5bd8"}
Nov 03 22:24:46.371787 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:46.371750    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 22:24:46.379783 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:46.379758    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 22:24:46.380315 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:46.380295    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rhpjv_6547c71f-71a2-40c9-912f-9283b6bdb560/console-operator/2.log"
Nov 03 22:24:46.388389 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:46.388369    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fvkn6_a295913f-3811-47d6-9e85-a76182e45649/ovn-acl-logging/0.log"
Nov 03 22:24:51.932139 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:51.932095    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gs8dn/must-gather-jq762" event={"ID":"5fb5b8ff-178f-4c9a-951a-dd228d57ef4b","Type":"ContainerStarted","Data":"8d0b5558b8505f7d654277ee103629c613256b4ec00bb23068d0cd6286b19e13"}
Nov 03 22:24:51.932668 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:51.932148    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gs8dn/must-gather-jq762" event={"ID":"5fb5b8ff-178f-4c9a-951a-dd228d57ef4b","Type":"ContainerStarted","Data":"41d7d65ba8e2338e0abe083444f0e286570d703c61d7c349fefcdc5c3775c9bf"}
Nov 03 22:24:52.045998 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:52.045944    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-gs8dn/must-gather-jq762" podStartSLOduration=2.734250265 podStartE2EDuration="9.045929691s" totalImagesPullingTime="6.311679426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:24:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:24:52.0433873 +0000 UTC m=+4206.537309174" watchObservedRunningTime="2026-11-03 22:24:52.045929691 +0000 UTC m=+4206.539851566"
Nov 03 22:24:53.923250 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:53.923193    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v5vrf_96fee653-58b4-46bf-9ed5-0d3c5a93e8a8/global-pull-secret-syncer/0.log"
Nov 03 22:24:54.500020 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:54.499993    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-27lr2_974ed564-6386-4e1e-9326-f14ccc36658c/connection-checker/0.log"
Nov 03 22:24:54.918310 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:54.918203    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-bhpkz_bbd3c724-e7e5-4382-b681-adf7fd8f302b/connection-checker/0.log"
Nov 03 22:24:55.340040 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:55.340014    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-sds64_52709fc1-b443-4a46-a95e-20cacc4ac4a0/connection-checker/0.log"
Nov 03 22:24:56.348169 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:56.348137    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cccr5_9046c34b-bb8f-4397-8b9d-7e2095add002/konnectivity-agent/0.log"
Nov 03 22:24:56.660073 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:56.659991    5053 scope.go:122] "RemoveContainer" containerID="120f55323c5e5667c59b4a7c2d647e3ee9a37dba162d9cabcd84028e6ca6ae46"
Nov 03 22:24:56.962278 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:56.962213    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngwtq" event={"ID":"9e988f30-c8ac-4357-9d38-a7720a906f06","Type":"ContainerStarted","Data":"9f08dddcca20e6d66dab3f632f9acc4932d96b24b12ca337d062ebec038957c7"}
Nov 03 22:24:56.962593 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:56.962569    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngwtq"
Nov 03 22:24:57.834115 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:24:57.834074    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal_8253f9611b0fc9c6ff64af8baa405814/haproxy/0.log"
Nov 03 22:25:05.295975 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:05.295939    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-96cd8748-ck6m2_a1461a40-86ef-4281-bb1f-b157e7b4e2f3/manager/0.log"
Nov 03 22:25:05.634284 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:05.634198    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-vmp59_36ec7139-4665-4a3c-ab8e-5ac33799ca58/manager/0.log"
Nov 03 22:25:05.876859 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:05.876819    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-kmgzq_1028ce52-b2d5-4d38-a449-b3396a58d4db/manager/0.log"
Nov 03 22:25:07.327810 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:07.327778    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-z5jk4_a3134aab-ba99-4e98-974a-dc26c5966860/server/0.log"
Nov 03 22:25:07.970790 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:07.970751    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngwtq"
Nov 03 22:25:08.314019 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:08.313935    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-h2zs5_53dadcdb-7f9d-4dd1-acb8-9321d2bf3ef3/manager/0.log"
Nov 03 22:25:08.427563 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:08.427535    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-pt78x_01bb8d54-6cba-4a24-ac55-4e15a54b87db/proxy/0.log"
Nov 03 22:25:10.927564 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:25:10.927527    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:25:11.282458 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:11.282429    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-96cd8748-ck6m2_a1461a40-86ef-4281-bb1f-b157e7b4e2f3/manager/0.log"
Nov 03 22:25:11.522527 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:11.522497    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-vmp59_36ec7139-4665-4a3c-ab8e-5ac33799ca58/manager/0.log"
Nov 03 22:25:11.760060 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:11.760026    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-kmgzq_1028ce52-b2d5-4d38-a449-b3396a58d4db/manager/0.log"
Nov 03 22:25:13.239772 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:13.239742    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-z5jk4_a3134aab-ba99-4e98-974a-dc26c5966860/server/0.log"
Nov 03 22:25:14.227881 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:14.227812    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-h2zs5_53dadcdb-7f9d-4dd1-acb8-9321d2bf3ef3/manager/0.log"
Nov 03 22:25:14.373515 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:14.373482    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-pt78x_01bb8d54-6cba-4a24-ac55-4e15a54b87db/proxy/0.log"
Nov 03 22:25:16.636742 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:16.636699    5053 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-43.us-west-1.compute.internal" podUID="8253f9611b0fc9c6ff64af8baa405814" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Nov 03 22:25:17.759375 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:17.759338    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-58c98c4485fcxmh_e8886b8b-7d2c-4b60-8843-651e85091077/istio-proxy/0.log"
Nov 03 22:25:19.054210 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:19.054166    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-tf4cx"]
Nov 03 22:25:19.062539 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:19.062514    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tf4cx"
Nov 03 22:25:19.116696 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:19.116664    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-tf4cx"]
Nov 03 22:25:19.163734 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:19.163688    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 03 22:25:19.163734 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:19.163721    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 03 22:25:19.164579 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:19.164552    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nvf9h\""
Nov 03 22:25:19.248455 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:19.248420    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d82efb7f-c92a-47d5-a88b-71aeba8d04ba-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-tf4cx\" (UID: \"d82efb7f-c92a-47d5-a88b-71aeba8d04ba\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tf4cx"
Nov 03 22:25:19.248683 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:19.248475    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d82efb7f-c92a-47d5-a88b-71aeba8d04ba-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-tf4cx\" (UID: \"d82efb7f-c92a-47d5-a88b-71aeba8d04ba\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tf4cx"
Nov 03 22:25:19.248683 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:19.248627    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnc7k\" (UniqueName: \"kubernetes.io/projected/d82efb7f-c92a-47d5-a88b-71aeba8d04ba-kube-api-access-vnc7k\") pod \"kuadrant-console-plugin-66c4fb575d-tf4cx\" (UID: \"d82efb7f-c92a-47d5-a88b-71aeba8d04ba\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tf4cx"
Nov 03 22:25:19.349770 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:19.349668    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnc7k\" (UniqueName: \"kubernetes.io/projected/d82efb7f-c92a-47d5-a88b-71aeba8d04ba-kube-api-access-vnc7k\") pod \"kuadrant-console-plugin-66c4fb575d-tf4cx\" (UID: \"d82efb7f-c92a-47d5-a88b-71aeba8d04ba\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tf4cx"
Nov 03 22:25:19.349770 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:19.349750    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d82efb7f-c92a-47d5-a88b-71aeba8d04ba-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-tf4cx\" (UID: \"d82efb7f-c92a-47d5-a88b-71aeba8d04ba\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tf4cx"
Nov 03 22:25:19.349770 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:19.349773    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d82efb7f-c92a-47d5-a88b-71aeba8d04ba-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-tf4cx\" (UID: \"d82efb7f-c92a-47d5-a88b-71aeba8d04ba\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tf4cx"
Nov 03 22:25:19.350523 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:19.350497    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d82efb7f-c92a-47d5-a88b-71aeba8d04ba-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-tf4cx\" (UID: \"d82efb7f-c92a-47d5-a88b-71aeba8d04ba\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tf4cx"
Nov 03 22:25:19.352645 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:19.352623    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d82efb7f-c92a-47d5-a88b-71aeba8d04ba-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-tf4cx\" (UID: \"d82efb7f-c92a-47d5-a88b-71aeba8d04ba\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tf4cx"
Nov 03 22:25:19.406358 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:19.406325    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnc7k\" (UniqueName: \"kubernetes.io/projected/d82efb7f-c92a-47d5-a88b-71aeba8d04ba-kube-api-access-vnc7k\") pod \"kuadrant-console-plugin-66c4fb575d-tf4cx\" (UID: \"d82efb7f-c92a-47d5-a88b-71aeba8d04ba\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tf4cx"
Nov 03 22:25:19.675323 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:19.675182    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tf4cx"
Nov 03 22:25:20.047202 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:25:20.047171    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd82efb7f_c92a_47d5_a88b_71aeba8d04ba.slice/crio-e7a04e497e8b691651d6418bec2cfd53ae5156d12e250c2fb7e83ff90661ee39 WatchSource:0}: Error finding container e7a04e497e8b691651d6418bec2cfd53ae5156d12e250c2fb7e83ff90661ee39: Status 404 returned error can't find the container with id e7a04e497e8b691651d6418bec2cfd53ae5156d12e250c2fb7e83ff90661ee39
Nov 03 22:25:20.049674 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:20.049580    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-75b555675f-kp6g9_5665aa62-30d5-4ced-8b5c-fedb882823cf/istio-proxy/0.log"
Nov 03 22:25:20.064273 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:20.064203    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-tf4cx"]
Nov 03 22:25:20.096574 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:20.096529    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tf4cx" event={"ID":"d82efb7f-c92a-47d5-a88b-71aeba8d04ba","Type":"ContainerStarted","Data":"e7a04e497e8b691651d6418bec2cfd53ae5156d12e250c2fb7e83ff90661ee39"}
Nov 03 22:25:20.694815 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:20.694784    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5ffbbbfcf7-99qbj_7ccca940-de12-451b-b920-d6bef516942f/router/0.log"
Nov 03 22:25:21.984570 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:21.984532    5053 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2"]
Nov 03 22:25:22.005628 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:22.005595    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2"
Nov 03 22:25:22.069788 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:22.069700    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ph99f\"/\"kube-root-ca.crt\""
Nov 03 22:25:22.081852 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:22.081818    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkstt\" (UniqueName: \"kubernetes.io/projected/b32d7004-b260-4e93-bc7b-cef22ae3eb4e-kube-api-access-mkstt\") pod \"perf-node-gather-daemonset-qs9s2\" (UID: \"b32d7004-b260-4e93-bc7b-cef22ae3eb4e\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2"
Nov 03 22:25:22.082042 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:22.081872    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b32d7004-b260-4e93-bc7b-cef22ae3eb4e-lib-modules\") pod \"perf-node-gather-daemonset-qs9s2\" (UID: \"b32d7004-b260-4e93-bc7b-cef22ae3eb4e\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2"
Nov 03 22:25:22.082042 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:22.081998    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b32d7004-b260-4e93-bc7b-cef22ae3eb4e-podres\") pod \"perf-node-gather-daemonset-qs9s2\" (UID: \"b32d7004-b260-4e93-bc7b-cef22ae3eb4e\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2"
Nov 03 22:25:22.082042 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:22.082037    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b32d7004-b260-4e93-bc7b-cef22ae3eb4e-proc\") pod \"perf-node-gather-daemonset-qs9s2\" (UID: \"b32d7004-b260-4e93-bc7b-cef22ae3eb4e\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2"
Nov 03 22:25:22.082218 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:22.082065    5053 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b32d7004-b260-4e93-bc7b-cef22ae3eb4e-sys\") pod \"perf-node-gather-daemonset-qs9s2\" (UID: \"b32d7004-b260-4e93-bc7b-cef22ae3eb4e\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2"
Nov 03 22:25:22.119258 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:22.108957    5053 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ph99f\"/\"default-dockercfg-vjzx4\""
Nov 03 22:25:22.119258 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:22.109025    5053 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ph99f\"/\"openshift-service-ca.crt\""
Nov 03 22:25:22.119258 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:22.112209    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2"]
Nov 03 22:25:22.188309 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:22.183261    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkstt\" (UniqueName: \"kubernetes.io/projected/b32d7004-b260-4e93-bc7b-cef22ae3eb4e-kube-api-access-mkstt\") pod \"perf-node-gather-daemonset-qs9s2\" (UID: \"b32d7004-b260-4e93-bc7b-cef22ae3eb4e\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2"
Nov 03 22:25:22.188309 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:22.183317    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b32d7004-b260-4e93-bc7b-cef22ae3eb4e-lib-modules\") pod \"perf-node-gather-daemonset-qs9s2\" (UID: \"b32d7004-b260-4e93-bc7b-cef22ae3eb4e\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2"
Nov 03 22:25:22.188309 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:22.183374    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b32d7004-b260-4e93-bc7b-cef22ae3eb4e-podres\") pod \"perf-node-gather-daemonset-qs9s2\" (UID: \"b32d7004-b260-4e93-bc7b-cef22ae3eb4e\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2"
Nov 03 22:25:22.188309 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:22.183406    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b32d7004-b260-4e93-bc7b-cef22ae3eb4e-proc\") pod \"perf-node-gather-daemonset-qs9s2\" (UID: \"b32d7004-b260-4e93-bc7b-cef22ae3eb4e\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2"
Nov 03 22:25:22.188309 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:22.183429    5053 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b32d7004-b260-4e93-bc7b-cef22ae3eb4e-sys\") pod \"perf-node-gather-daemonset-qs9s2\" (UID: \"b32d7004-b260-4e93-bc7b-cef22ae3eb4e\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2"
Nov 03 22:25:22.188309 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:22.183598    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b32d7004-b260-4e93-bc7b-cef22ae3eb4e-sys\") pod \"perf-node-gather-daemonset-qs9s2\" (UID: \"b32d7004-b260-4e93-bc7b-cef22ae3eb4e\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2"
Nov 03 22:25:22.188309 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:22.183617    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b32d7004-b260-4e93-bc7b-cef22ae3eb4e-podres\") pod \"perf-node-gather-daemonset-qs9s2\" (UID: \"b32d7004-b260-4e93-bc7b-cef22ae3eb4e\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2"
Nov 03 22:25:22.188309 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:22.183665    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b32d7004-b260-4e93-bc7b-cef22ae3eb4e-proc\") pod \"perf-node-gather-daemonset-qs9s2\" (UID: \"b32d7004-b260-4e93-bc7b-cef22ae3eb4e\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2"
Nov 03 22:25:22.188309 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:22.183679    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b32d7004-b260-4e93-bc7b-cef22ae3eb4e-lib-modules\") pod \"perf-node-gather-daemonset-qs9s2\" (UID: \"b32d7004-b260-4e93-bc7b-cef22ae3eb4e\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2"
Nov 03 22:25:22.240816 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:22.240745    5053 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkstt\" (UniqueName: \"kubernetes.io/projected/b32d7004-b260-4e93-bc7b-cef22ae3eb4e-kube-api-access-mkstt\") pod \"perf-node-gather-daemonset-qs9s2\" (UID: \"b32d7004-b260-4e93-bc7b-cef22ae3eb4e\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2"
Nov 03 22:25:22.341439 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:22.341401    5053 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2"
Nov 03 22:25:22.739782 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:22.739756    5053 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2"]
Nov 03 22:25:22.742217 ip-10-0-128-43 kubenswrapper[5053]: W1103 22:25:22.742179    5053 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb32d7004_b260_4e93_bc7b_cef22ae3eb4e.slice/crio-72e995326abc8ed628c321743a9d2c50086e872ec7c1c78862e3ca06b395a057 WatchSource:0}: Error finding container 72e995326abc8ed628c321743a9d2c50086e872ec7c1c78862e3ca06b395a057: Status 404 returned error can't find the container with id 72e995326abc8ed628c321743a9d2c50086e872ec7c1c78862e3ca06b395a057
Nov 03 22:25:23.125822 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:23.125735    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2" event={"ID":"b32d7004-b260-4e93-bc7b-cef22ae3eb4e","Type":"ContainerStarted","Data":"01b6c38517b712a960758f9685336c209f3e243dc0065014345d60a387f34c47"}
Nov 03 22:25:23.125822 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:23.125809    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2" event={"ID":"b32d7004-b260-4e93-bc7b-cef22ae3eb4e","Type":"ContainerStarted","Data":"72e995326abc8ed628c321743a9d2c50086e872ec7c1c78862e3ca06b395a057"}
Nov 03 22:25:23.126727 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:23.126702    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2"
Nov 03 22:25:23.239752 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:23.239668    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2" podStartSLOduration=2.23965038 podStartE2EDuration="2.23965038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:23.23878715 +0000 UTC m=+4237.732709035" watchObservedRunningTime="2026-11-03 22:25:23.23965038 +0000 UTC m=+4237.733572256"
Nov 03 22:25:26.953142 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:26.953109    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-jnjhr_bcdcbe12-1bff-4947-ab42-909bd405e7b1/manager/0.log"
Nov 03 22:25:27.151307 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:27.151269    5053 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tf4cx" event={"ID":"d82efb7f-c92a-47d5-a88b-71aeba8d04ba","Type":"ContainerStarted","Data":"16b3e7916e3ce8767d5bbcfd5a7b7d585068e544e1d536c6eb9e4684269b3818"}
Nov 03 22:25:27.213980 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:27.213896    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-7twr6_79d0e65f-9499-4fd9-8b3c-6f70bd3468f1/manager/0.log"
Nov 03 22:25:27.264282 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:27.264214    5053 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tf4cx" podStartSLOduration=2.226805984 podStartE2EDuration="8.264200622s" totalImagesPullingTime="6.037394638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:27.263057191 +0000 UTC m=+4241.756979067" watchObservedRunningTime="2026-11-03 22:25:27.264200622 +0000 UTC m=+4241.758122497"
Nov 03 22:25:27.426621 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:27.426586    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-tf4cx_d82efb7f-c92a-47d5-a88b-71aeba8d04ba/kuadrant-console-plugin/0.log"
Nov 03 22:25:27.640603 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:27.640529    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-ngwtq_9e988f30-c8ac-4357-9d38-a7720a906f06/manager/1.log"
Nov 03 22:25:27.642639 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:27.642608    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-ngwtq_9e988f30-c8ac-4357-9d38-a7720a906f06/manager/2.log"
Nov 03 22:25:27.994852 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:27.994817    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-qrdkr_757361e7-9919-408f-9377-317923c5ffa5/manager/0.log"
Nov 03 22:25:30.149784 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:30.149753    5053 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-qs9s2"
Nov 03 22:25:40.931640 ip-10-0-128-43 kubenswrapper[5053]: E1103 22:25:40.931595    5053 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9vxl" containerName="extractor"
Nov 03 22:25:44.296893 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:44.296862    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-58c98c4485fcxmh_e8886b8b-7d2c-4b60-8843-651e85091077/istio-proxy/0.log"
Nov 03 22:25:45.849925 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:45.849901    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5745664b78-brxc8_3b9b9bb9-ab83-4ab3-9de3-acc59e05c900/manager/0.log"
Nov 03 22:25:46.058874 ip-10-0-128-43 kubenswrapper[5053]: I1103 22:25:46.058847    5053 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5745664b78-brxc8_3b9b9bb9-ab83-4ab3-9de3-acc59e05c900/copy-manifests/0.log"